Developer guide

Adding an assignment

A new assignment needs two things: a rubric YAML in rubrics/ and a grader class in assignments/ that defines a check_<rubric_id> method for each item. assignments.base takes care of dispatch, the execution check and error handling.

1. Write the rubric#

Start from the closest existing rubric. Decide first which situation the assignment is in, because that determines what the checks can compare against:

Every student uses the same distributed data (like Assignment 3): write the expected values into the rubric and match against them.
Students fetch their own data (like Assignment 1): check each step against the student's own data, never against a stored number.
The data is never handed in (like Assignment 2): grade the outputs saved in the notebook.

# rubrics/hw4.yaml
assignment:
  id: hw4
  name: "Assignment 4: Street Networks"
  version: HW4-v1
  total_points: 100

settings:
  score_rounding: 0.5

discovery:
  notebook_name_hints: ["assignment-4", "assignment4", "hw4"]
  ignore_name_patterns: ["checkpoint", "untitled", "scratch", "solution"]

data:
  required: true
  description: The OSM extract distributed with the assignment.

rubric:
  - id: notebook_execution
    name: Notebook executes successfully
    points: 10
    type: execution
    config:
      partial_credit_ratio: 0.4

  - id: graph_loaded
    name: Street graph loaded with osmnx
    points: 90
    type: deterministic
    config:
      min_nodes: 1000
      wrong_crs_credit_ratio: 0.5

Until a check exists, its item is returned as manual_review, so the rubric loads and runs from the start.

2. Write the grader class#

# assignments/hw4.py
"""Assignment 4 autograder — street networks."""

from __future__ import annotations

from grader import inspection
from grader.rubric import RubricItem
from grader.models import RubricItemResult

from .base import (
    STATUS_FAIL, STATUS_NOT_FOUND, STATUS_PARTIAL, STATUS_PASS,
    AssignmentGrader, GradingContext, register,
)


@register
class HW4Grader(AssignmentGrader):
    assignment_id = "hw4"          # must match assignment.id in the YAML

    # notebook_execution is inherited from AssignmentGrader.

    def check_graph_loaded(self, ctx: GradingContext, item: RubricItem) -> RubricItemResult:
        if not ctx.probe_ok:
            return self.no_probe_result(item, ctx)

        min_nodes = int(item.config.get("min_nodes", 1000))
        frames = [p for p in inspection.dataframes(ctx.probe)
                  if p.get("rows", 0) >= min_nodes]
        if not frames:
            return self.result(item, 0, STATUS_NOT_FOUND, 1.0,
                               "No node table of the expected size was found.")
        if len(frames) > 1:
            # Ambiguity lowers confidence, which sends the item to review.
            return self.result(item, item.points, STATUS_PASS, 0.75,
                               "Several plausible node tables were found.",
                               {"candidates": len(frames)})
        return self.result(item, item.points, STATUS_PASS, 0.98,
                           "Street graph loaded.", {"rows": frames[0]["rows"]})

Then register the module so importing the package loads it. Add it to the imports in assignments/__init__.py and to the import inside get_grader() and registered_assignments() in assignments/base.py:

from . import hw1, hw2, hw3, hw4  # noqa: F401  (registers the assignment graders)

The check contract#

Rule	Why
Return exactly one `RubricItemResult`, built with `self.result(...)`	`result()` rounds to `score_rounding` and clamps to `[0, points]`
Put the reason in `evidence`	The TA sees it on the student page. A deduction a TA can't explain has to be worked out again by hand.
Name the mistake in `feedback`	"Wrong" isn't useful feedback. "Latitude and longitude are swapped" is.
Lower `confidence` when choosing between readings	Anything below the threshold goes to review. 1.0 means the result was matched exactly.
Read thresholds and ratios from `item.config`	Policy belongs in the YAML, so it can be fixed without touching Python
Don't worry about raising	`grade()` turns an exception into a `manual_review` item, so a broken check never loses a submission
Never depend on variable names	Search the probe payload by shape and content using `grader.inspection`

What a check can see#

Every check receives a GradingContext:

Attribute	Contents
`ctx.analysis`	`NotebookAnalysis`: static analysis of the submitted source (imports, functions, read calls, paths, markdown cells)
`ctx.execution`	`ExecutionRecord`: success, timeout, error cell, traceback
`ctx.probe`	The probe's JSON description of the kernel namespace, or `None` if the kernel never reported
`ctx.workdir`	The submission's working directory, including the executed notebook
`ctx.qualitative_grader`	The LLM grader, or a disabled one. Call `.grade(question, response, rubric)`.
`ctx.cache`	Scratch space shared between checks on one submission

To grade from saved outputs instead of (or as well as) the re-run, use grader.answers for printed values and grader.charts for chart specs.

Hidden tests#

Declare the item as type: hidden_test with a selector and cases (see Hidden-test config). AssignmentGrader.hidden_test_specs() sends only the selector and the inputs to the probe. The probe calls the student's function in-kernel and reports what came back, and your check compares that against the expected values, which stay on the host.

3. Add tests and demo submissions#

Follow the pattern in tests/test_hw3.py and examples/make_hw3_submissions.py: one synthetic submission that does everything right, one that makes the near-misses students really make, and one that crashes. Assert that the correct one scores full marks (or close, on judgement items) and that each near-miss is named in the feedback.

Never commit student work

Reproduce the failure patterns you see in synthetic fixtures. No student file, name or identifier belongs in the repository.