User guide

How grading works

Every submission goes through the same pipeline. Each stage is built so that a failure produces a flag for a person rather than a silent zero.

0. The data comes from you, not the students#

Students hand in a notebook and nothing else, so the grader supplies the data. You upload the file once (for Assignment 3, all five files) in the sidebar or with --data on the CLI. Before each submission runs, the grader places the files at:

• every relative path the notebook actually reads from, parsed out of the notebook, including paths held in a variable rather than passed inline, and
• the conventional data/<name> and <name> locations, for notebooks that build the path dynamically.

So a notebook reading data/zillow.csv and one reading data/Zip_zhvi_uc_sfrcondo_tier_0.33_0.67_sm_sa_month.csv both find their file, and the class doesn't have to follow a naming convention. When several files are supplied, each target path gets the file whose name is closest to it.

• Files are hard-linked, not copied, so fifty submissions use one copy of a 117 MB extract rather than fifty.
• A student's own file is never overwritten. Set data.prefer_student_files: true in the rubric to grade each student on their own copy instead.
• Paths that escape the working directory (../, absolute paths, Windows drive letters) are refused.
• A notebook that reads from a URL is redirected to the supplied local file.

Everyone is then graded against the same data, which makes results comparable across the class. A notebook that hardcodes /Users/me/Desktop/... still fails, as it should, because that path cannot be supplied.

1. Discovery#

Notebook filenames are not standardised, so the grader scores each candidate notebook on name hints (from the rubric's discovery.notebook_name_hints), directory depth and size. Names matching ignore_name_patterns (checkpoint, untitled, scratch, solution and so on) are ruled out. When two files look equally plausible, the submission is flagged, not guessed: it goes to the review queue as multiple_notebooks.

Both submission layouts are supported:

submissions/                         canvas_export.zip
├── student_001/                     ├── doejane_12345_67890_assignment1.ipynb
│   └── assignment-1.ipynb           ├── smithj_late_23456_78901_hw1.ipynb
├── student_002/                     └── ...
│   ├── assignment-1.ipynb
│   └── scratch.ipynb
└── ...

Nested archives inside a submission are expanded, and a single wrapping directory is unwrapped. The code is in grader.discovery.

2. Execution#

Each submission is copied into its own working directory. The notebook is instrumented with probe cells, and a subprocess runs it. By default that subprocess runs inside a container with no network, capped memory and CPU, a read-only root filesystem and no host environment variables. See Docker sandbox.

Errors do not stop grading. allow_errors is on, so the notebook runs to the end, the first failing cell and its traceback are recorded, and every check that can still run does. A failed notebook never becomes an automatic zero. The execution item itself awards partial credit for how far the notebook got: failing at the last cell is worth more than failing at the first.

3. Inspection without variable names#

Students don't use the solution's variable names, so the probe describes the kernel's namespace rather than looking anything up by name. For every object it records:

Some summaries are deliberately compact. 384 GEOIDs are far more than the value-sample cap, so instead of the list the probe sends a histogram of their first five characters. That answers "is every row in Philadelphia County?" in a few bytes.

The assignment grader then looks for objects that look like the expected result (right ZIP prefix, right row count, right roles) and reports how many plausible candidates it saw. One unique match gives high confidence, two give 0.75, and more give 0.40.

The probe source is in grader.probe_runtime. It runs only inside the student's kernel, never on the host, and everything it returns is a JSON summary.

4. Hidden tests#

calculate_percent_increase(group_df) takes a DataFrame, so it can't be tested with literal arguments. Instead the probe reads the column names and date dtype of the student's own tidy frame, builds a synthetic group frame from interpolated anchor points using those names, and calls the function with it. The expected answer follows from the anchors, so it is known whatever data the student loaded.

The anchors deliberately imply a different answer for the first and last rows than for the two dates named in the assignment. That lets the grader identify the most common wrong implementation by name instead of just marking it wrong:

Two readings: saved outputs and the re-run#

Assignments 2 and 3 can't rely on a re-run alone. Assignment 2's data is never handed in, and Assignment 3 reads five files from student-chosen paths. If placement misses, the notebook dies at the first read_file. The students' own runs didn't have that problem, and the notebooks they handed in carry what those runs printed:

In  [3]: len(phl)
Out [3]: 384

grader.answers reads those saved answers, and grader.charts reads the Vega-Lite specs and bokeh buffers saved with each chart. Each Assignment 3 item is graded on two independent readings: what the submitted notebook shows and what this run produced. A step counts as done if either source shows the right answer. Two rules keep this honest:

• A number must be shown to count. Cell values inside a head() repr are not answers.
• Source code alone is never full marks. A step the source clearly performs but never displays earns 70% and gets a look from a person.

5. Confidence and review#

A submission's confidence is its lowest item confidence, because the weakest item decides how far the grade can be trusted. The policy is implemented in grader.scoring.apply_review_policy.

Item statuses#

Every rubric item comes back with one of these statuses. The UI never uses colour alone: each status also has an icon and a word.

Scores and rounding#

Partial credit is rounded to the rubric's settings.score_rounding increment (0.5 by default), so a TA never has to explain a score of 14.28. Scores are clamped to [0, points]. A check that raises an exception never loses the submission: the item comes back as manual_review with confidence 0 and the error in its evidence.