User guide

Rubrics

Everything the grader expects lives in rubrics/*.yaml: expected values, tolerances, hidden-test anchors, and credit ratios for common near-misses. To change the grading, edit the YAML and press Regrade. No Python changes are needed.

Before grading a real class#

File layout#

A rubric has six top-level blocks. Only assignment.id and at least one rubric item are required.

assignment:            # who this rubric is for
  id: hw1              # required; selects the grader class registered under this id
  name: "Assignment 1: The Donut Effect"
  version: HW1-v2      # stored with every session
  total_points: 100    # must equal the sum of the item points
  description: >
    Load the Zillow ZHVI data, trim to Philadelphia, ...

settings: {...}        # grading behaviour
discovery: {...}       # how to find the notebook in a submission
data: {...}            # data the instructor supplies
probe: {...}           # hints for the in-kernel probe
rubric: [...]          # the graded items

load_rubric() validates the file and raises RubricError on a missing id, an empty rubric, a duplicate item id, an unknown item type, or item points that don't add up to total_points.

assignment#

settings#

discovery#

discovery:
  notebook_name_hints: ["assignment-1", "assignment1", "hw1", "homework1", "musa"]
  ignore_name_patterns: ["checkpoint", "untitled", "scratch", "solution", "draft", "copy of"]

Hints raise a notebook's score when it is chosen from a submission. Notebooks matching an ignore pattern are skipped. See Discovery.

data#

data:
  required: true                     # refuse to run without a data file (HW1)
  optional: false                    # offer the data upload without requiring it (HW3)
  description: >                     # shown in the sidebar and in preflight warnings
    Zillow ZHVI by ZIP code, e.g. Zip_zhvi_uc_sfrcondo_tier_0.33_0.67_sm_sa_month.csv
  expected_files: []                 # HW3 lists its five; preflight names any missing
  fallback_locations: ["data", ""]   # conventional places to put each file
  max_placements: 12                 # cap on target paths per submission (HW3: 40)
  prefer_student_files: false        # grade on the student's own file if they sent one

probe#

Hints the in-kernel probe uses to recognise columns, plus limits on how much it reports. Hints are matched against column names, case-insensitively.

probe:
  id_column_hints: ["zipcode", "zip_code", "zip", "regionname", "postal"]
  date_column_hints: ["date", "month", "year", "period", "time", "variable"]
  value_column_hints: ["zhvi", "value", "home_value", "price", "index"]
  max_head_rows: 5          # default 5
  max_unique_values: 120    # default 80

rubric items#

Each item maps to one check_<id> method on the assignment's grader class. An item without a matching method comes back as manual_review with the note "add a check_<id> method", so a rubric can be written before its checks.

rubric:
  - id: notebook_execution           # unique; selects check_notebook_execution
    name: Notebook executes successfully
    points: 10
    type: execution                  # see the table below
    description: The notebook runs top to bottom in a clean environment.
    config:                          # free-form; read by the check method
      partial_credit_ratio: 0.4

Item types#

Hidden-test config#

Two call styles are supported. scalar_args passes literal arguments. group_frame asks the probe to build a DataFrame from interpolated anchor points, using the student's own column names.

- id: percent_increase_function
  points: 20
  type: hidden_test
  config:
    type: group_frame
    selector:                        # how to find the student's function
      name_hints: ["percent", "increase", "calculate", "change", "growth"]
      arity: 1
    start_date: "2020-03-31"
    end_date: "2022-03-31"
    frame:
      id_column_hints: ["regionname", "zipcode", "zip"]
      date_column_hints: ["date", "month", "variable"]
      value_column_hints: ["zhvi", "value"]
      default_id_column: "RegionName"
      default_date_column: "date"
      default_value_column: "ZHVI"
    cases:
      - label: "100 → 150 (Mar 2020 → Mar 2022)"
        anchors: [["2019-01-31", 80], ["2020-03-31", 100], ["2022-03-31", 150], ["2023-12-31", 200]]
    rel_tol: 0.001
    abs_tol: 0.001
    fraction_form_credit_ratio: 0.6  # returned 0.5 instead of 50
    first_last_credit_ratio: 0.5     # used the first and last rows
    stub_credit_ratio: 0.0           # left as the template

For scalar_args, each case has args and optional kwargs and label.

Credit ratios#

Most item configs end in *_credit_ratio or *_penalty keys. These are the named near-misses: each is the fraction of the item's points awarded when the check recognises that particular mistake. They are policy, so tune them freely. The assignment grader reference documents which keys each check reads.

Validating a rubric#

python cli.py rubrics

A rubric that fails to parse is left out of python cli.py rubrics and the sidebar. Load it directly to see the error:

from grader.rubric import load_rubric
load_rubric("rubrics/hw4.yaml")   # raises RubricError with the reason