Introduction

Getting started

Install the grader, build the sandbox image, and grade a demo class in about ten minutes.

Requirements#

• Python 3.10 or newer. CI tests 3.10 through 3.13.
• Docker, recommended. Student code is untrusted, and the default execution mode runs every notebook in an isolated container. Without Docker you can still use --mode local, but notebooks then run on your machine with no isolation.
• For Assignment 3's demo submissions only: the geospatial stack (geopandas, rasterio, hvplot and the rest). The sandbox image already includes all of it.

Install#

1. Install the Python dependencies

   pip install -r requirements.txt

   Or install the package in editable mode with the development tools (pytest, ruff):

   pip install -e ".[dev]"

2. Build the grading container

   Do this once per semester, and again whenever the sandbox packages change.

   docker build -t musa-grader:latest -f docker/Dockerfile .

3. Check that the rubrics load

   python cli.py rubrics

   This should list hw1, hw2 and hw3 with their YAML paths.

Grade a demo class#

The repository ships generators for synthetic submissions covering the cases a TA usually sees. The submissions contain notebooks only, with no data, which is how the class hands work in.

Assignment 1#

python examples/make_example_data.py && python examples/make_example_submissions.py

This builds seven submissions: two correct, one with plausible mistakes, one with two notebooks, one that crashes partway, one with no notebook, and one untouched template.

python cli.py grade examples/submissions --mode docker --data examples/data/zillow_zhvi.csv

Assignment 2#

python examples/make_hw2_data.py && python examples/make_hw2_submissions.py

These four submissions are run against the data and saved with their outputs, then handed in without the data. One does everything including the extra credit, one makes the usual near-misses, one hit an error in the student's own run, and one was never run. No data file is needed for grading: every notebook will fail to execute, and that is expected, because the rubric grades the outputs the students saved.

python cli.py grade examples/hw2_submissions --rubric rubrics/hw2.yaml

Assignment 3#

python examples/make_hw3_submissions.py
python cli.py grade examples/hw3_submissions --rubric rubrics/hw3.yaml

Expected scores: 99.5, 68.5, 34.5 and 10 out of 100. The correct submission misses 0.5 because of the ceiling on the styling item: no submission reaches full marks on a judgement item until a person has looked at it.

To have the grader re-run each notebook and confirm what it shows, unzip the data and supply all five files:

unzip assignment_template/assignment3-data.zip -d /tmp/hw3
python cli.py grade examples/hw3_submissions --rubric rubrics/hw3.yaml \
  --data /tmp/hw3/assignment3-data/PA-tracts.geojson \
  --data /tmp/hw3/assignment3-data/li_violations.csv \
  --data /tmp/hw3/assignment3-data/landsat8_philly.tif \
  --data /tmp/hw3/assignment3-data/City_Limits.geojson \
  --data /tmp/hw3/assignment3-data/ppr_tree_canopy_points_2015.geojson

Start the web app#

streamlit run app.py

Then, in the sidebar:

1. Pick Assignment 1.
2. Upload the assignment data file. For the demo, use examples/data/zillow_zhvi.csv.
3. Load examples/submissions, or upload a Canvas ZIP.
4. Choose an execution mode and press Run Grader.

See TA workflow for a tour of each page.

Run everything in Docker#

To move the grader to a new machine without setting up Python, build the sandbox image and start the app with Compose. Docker is the only requirement.

docker build -t musa-grader:latest -f docker/Dockerfile .
docker compose up -d --build

Then open http://localhost:8501. See Docker sandbox for how the app container starts the sandbox containers.

Next steps#