User guide

Assignments

The dataset decides what can be checked. Each assignment gives the grader a different situation, and the rubric is designed around it.

Assignment 1: The Donut Effect#

rubrics/hw1.yamlassignments/hw1.pydata required

Students download their own Zillow ZHVI extract, and Zillow revises and extends that file every month. Row counts, ZIP counts and final percentages therefore differ legitimately between submissions, so the grader never compares against a stored answer. It asks whether each step in the instructions worked, and checks it against the student's own data:

The only fixed values in the rubric are the ones the assignment itself fixes: the Center City ZIP list printed in the student template, and the two comparison dates (2020-03-31 and 2022-03-31). A regression test confirms this: the same notebook graded against a shorter, smaller, rescaled ZHVI file still scores 100/100.

Assignment 2: Exploratory Visualization#

rubrics/hw2.yamlassignments/hw2.pyno data

Students pick their own dataset and hand in only the notebook. The grader has no copy of the data, so it cannot re-run anything: every submission dies at read_csv. The content of a chart was never checkable, and this assignment doesn't ask for it. It asks whether the student can produce the required elements. So the rubric splits the assignment in two.

Facts, graded automatically (70 points)#

An Altair chart compiles to a Vega-Lite specification, and that specification is stored in the notebook's own saved output. The grader reads mean/count/bin transformations, interval brushes and transform_filter() cross-filters out of the spec instead of guessing from the source text. That is how it tells apart the three things students most often confuse:

Judgement, graded by a person (30 points)#

The assignment grades the matplotlib chart on "colour choices and clarity" and asks for written reasoning under each chart. The quality of a picture and the substance of a paragraph can't be checked by a machine. Those items always come back as manual_review, whatever they score. Each carries a provisional score built from real signals (labelling and colour calls extracted from the source, word counts, the position of each markdown cell) plus the text itself. A TA confirms or adjusts a number instead of starting from a blank box.

The grader records, per chart, where the evidence came from. If an instructor does supply the dataset, a freshly executed chart replaces the saved one automatically and confidence rises from 0.85 to 0.95.

The extra-credit dashboard is worth 0 points in the rubric so that it never inflates the maximum. When it is found, the item is flagged with the suggested bonus. A manual override on it adds to the student's total without changing what they are graded out of.

Assignment 3: Evictions and the NDVI#

rubrics/hw3.yamlassignments/hw3.pydata optional

Assignment 3 hands out assignment3-data.zip, and everyone works from it. That one difference changes the whole design: for the first time the right answers exist, so the grader checks results rather than inferring intent.

Every value was recomputed under the variations students legitimately produce (crop=True and crop=False, masked and filled arrays, sjoin(how="inner") and how="left") before it was written into the rubric. Where a step has two defensible answers, both are listed rather than one being called wrong. For example, the optional unstack(fill_value=0).stack() fill gives 5,535 groups instead of 4,000.

Because the answers are fixed, a matched expected value scores confidence 1.0. Grading combines two readings, the saved outputs and the re-run, so the data is optional:

Supply the data and 19 of the 22 items come back confirmed by both readings. Leave it out and the grade comes from the submitted notebook alone, at one confidence step lower (0.9 rather than 1.0).

Plots are graded on the objects#

Two thirds of Part 1 uses hvplot, and an hvplot call returns a holoviews object that knows what it is:

Named mistakes, not just wrong answers#

Where the 100 points go#

Execution is 10 points, Part 1 is 58 across thirteen steps, and Part 2 is 32 across eight. Within each part, the weight follows how much the step matters: the melt, the spatial join and the NDVI masking are where the assignment's teaching happens, while the loading steps are hard to get wrong and cheap to fix.

Assignments 4–6#

Their student templates are in assignment_template/, but they are not graded yet. Each needs a rubric YAML and a grader class. See Adding an assignment.