API reference · grader/answers.py · 659 lines

`grader.answers`

Answers read out of a notebook's own saved outputs (design.md §23, §25).

Why this exists

Assignment 3 has an answer key — everyone runs the same zip — so the grader knows that Philadelphia has 384 tracts and that the median NDVI inside the city limits is 0.2025. The question is where to see the student's version of those numbers.

Re-running the notebook is one answer, and it is the strongest one when it works. But it is also the fragile one, and on this assignment it is fragile in a way that is nobody's fault: five data files instead of one, read from whatever path each student happened to use, sometimes built with os.path.join or a variable the grader cannot resolve. When the placement misses, the notebook dies at the first read_file and every step after it is invisible — a submission that was entirely correct grades as if nothing happened.

The student's own run does not have that problem. They ran it with their data in the right place, and the notebook they handed in carries what it printed:

In  [3]: len(phl)
Out [3]: 384

That 384 is the answer to 1.1.2, sitting in the file. So is 34108, and (368606317.0763259, 462551418.69179374), and Median NDVI, city: 0.2025. This module harvests them, and the checks in assignments/hw3.py accept either source: the step counts as done if the executed run shows it or the submitted notebook does. Execution stops being load-bearing and becomes corroboration.

What can be read, and what it is worth

Output	What it yields
`print(...)`	every number in the text — deliberately shown
a bare final expression	every number, when it is short enough to be a result rather than a table
a DataFrame repr	`5376 rows x 4 columns`, the column names, and nothing else — the cell values of a `head()` are not answers, and harvesting them invents matches
`.describe()`	count / mean / median / min / max
`.info()`	row count, column names, dtypes
an hvplot chart	`:HoloMap [year]` — the object's own repr, which says whether `dynamic=False` was passed, plus the widget's option list from the embedded HTML
matplotlib	`<Figure size 800x450 with 1 Axes>` and the PNG

A saved output is the student's claim about their own run, not a fact the grader established, so a check that rests on one alone says so and grades at lower confidence — the same caveat grader/charts.py records for a chart seen only in the submitted file. Outputs can be stale, and they can be edited by hand.

Summary

Name	Kind	Description
`HoloviewsRepr`	class	One holoviews object, as its own repr describes it.
`CellAnswers`	class	Everything one code cell's saved output states.
`NotebookAnswers`	class	Every answer a notebook shows, with where each one was seen.
`read_answers()`	function	Harvest one notebook's saved outputs.
`collect_answers()`	function	Answers from both notebooks, merged cell by cell.

Constants

Name	Value	Notes
`SOURCE_EXECUTED`	`'executed'`
`SOURCE_SUBMITTED`	`'submitted'`
`SOURCE_MERGED`	`'merged'`
`SOURCE_NONE`	`'none'`
`MAX_SCALAR_CHARS`	`6000`	A result short enough to be a result. Tables are excluded by their own markers rather than by length (a `head()` repr carries an HTML table, `.info()` and `.describe()` have shapes of their own), so this only has to rule out the pathological case — a cell that dumps a whole file. It is generous because `landsat.count, landsat.crs` prints a full WKT block, and the EPSG code inside that block is the answer.
`NUMBER_RE`	`re.compile('-?\\d+(?:\\.\\d+)?(?:[eE][-+]?\\d+)?')`
`SHAPE_TEXT_RE`	`re.compile('\\[(\\d[\\d,])\\s+rows?\\s+x\\s+(\\d[\\d,])\\s+columns?\\]')`	pandas prints its shape under a truncated frame: "[5376 rows x 4 columns]" in text and "<p>5376 rows × 4 columns</p>" in HTML (a multiplication sign, not x).
`SHAPE_HTML_RE`	`re.compile('(\\d[\\d,])\\srows?\\s[x×]\\s(\\d[\\d,])\\scolumns?')`
`ENTRIES_RE`	`re.compile('(\\d[\\d,]*)\\s+entries')`	`.info()`: "RangeIndex: 5376 entries, 0 to 5375"
`INFO_COLUMN_RE`	`re.compile('^\\s\\d+\\s+(\\S+)\\s+\\d[\\d,]\\s+non-null', re.M)`	`.info()` column table: " 0 GEOID 5376 non-null str"
`DESCRIBE_RE`	`re.compile('^\\s(count\|mean\|std\|min\|25%\|50%\|75%\|max)\\s+(-?\\d+(?:\\.\\d+)?(?:[eE][-+]?\\d+)?)\\s$', re.M)`	`.describe()` rows, in the text repr.
`FIGURE_RE`	`re.compile('<Figure size (\\d+)x(\\d+) with (\\d+) Axes?>')`
`HTML_TABLE_RE`	`re.compile('<table\\b', re.I)`
`HTML_HEADER_RE`	`re.compile('<th[^>]*>([^<]{1,60})</th>', re.I)`
`HOLOVIEWS_LINE_RE`	`re.compile('^(?P<indent>\\s)(?:\\.(?P<path>\\S+)\\s+)?:(?P<kind>[A-Za-z]\\w)(?:\\s+\\[(?P<kdims>[^\\]])\\])?(?:\\s+\\((?P<vdims>[^)])\\…`	The repr holoviews gives every object: :HoloMap [year] :Polygons [x,y] (evictions,GEOID) :Layout .Polygons.I :Polygons [x,y] (evictions)
`WIDGET_RE`	`re.compile('"title":"(?P<title>[^"]{1,80})","options":\\[(?P<options>[^\\]]{0,8000})\\]\|"options":\\[(?P<options2>[^\\]]{0,8000})\\],"title…`	The widget Panel renders beside a HoloMap, in the embedded bokeh document.
`QUOTED_RE`	`re.compile('"((?:[^"\\\\]\|\\\\.){0,200})"')`
`NDARRAY_RE`	`re.compile('"array":\\{"type":"bytes","data":"(?P<data>[A-Za-z0-9+/=]{4,200000})"\\},"shape":\\[(?P<length>\\d{1,9})\\],"dtype":"(?P<dtype>…`	Bokeh writes a chart's numeric columns into the notebook as a base64 (usually gzipped) buffer. This is where "the numbers actually plotted" live once the kernel is gone.
`STRUCT_CODES`	`{'float64': ('d', 8), 'float32': ('f', 4), 'int64': ('q', 8), 'int32': ('i', 4), 'int16': ('h', 2), 'int8': ('b', 1), 'uint64': ('Q', 8), '…`
`IMAGE_MIMES`	`('image/png', 'image/jpeg', 'image/svg+xml')`
`WIDGET_KINDS`	`('HoloMap', 'GridSpace', 'NdLayout')`	Containers whose whole point is one chart per value of a dimension.
`LAZY_KINDS`	`('DynamicMap',)`	The same thing, never evaluated: what hvplot returns without `dynamic=False`.
`LAYOUT_KINDS`	`('Layout', 'NdLayout')`

Classes

class HoloviewsRepr #[source]

dataclass

One holoviews object, as its own repr describes it.

Fields

Name	Type	Default	Notes
`kind`	`str`	required
`kdims`	`list[str]`	`field(default_factory=list)`
`vdims`	`list[str]`	`field(default_factory=list)`
`children`	`list[str]`	`field(default_factory=list)`
`frame_keys`	`list[str]`	`field(default_factory=list)`	Option list of the widget beside it, read from the embedded bokeh document.
`widget_dimension`	`str`	`''`
`values`	`dict[str, float]`	`field(default_factory=dict)`	The numbers the chart actually draws, decoded from that same document.
`code_cell`	`int`	`0`
`source`	`str`	`SOURCE_NONE`

Properties

property is_widget: bool #[source]: No description.

property is_lazy: bool #[source]: A DynamicMap: the widget exists but no frame was ever computed.

property is_layout: bool #[source]: No description.

property element_types: list[str] #[source]: No description.

Methods

summary() -> dict[str, Any] #[source]: No description.

class CellAnswers #[source]

dataclass

Everything one code cell's saved output states.

Fields

Name	Type	Default	Notes
`code_cell`	`int`	required
`cell_index`	`int`	required
`source_code`	`str`	required
`text`	`str`	`''`
`shown`	`list[float]`	`field(default_factory=list)`	Numbers the cell deliberately showed: printed, or returned as a short result. Never cell values scraped out of a table repr.
`shapes`	`list[tuple[int, int]]`	`field(default_factory=list)`
`row_counts`	`list[int]`	`field(default_factory=list)`
`columns`	`list[str]`	`field(default_factory=list)`
`describe`	`dict[str, float]`	`field(default_factory=dict)`
`holoviews`	`list[HoloviewsRepr]`	`field(default_factory=list)`
`figures`	`int`	`0`
`images`	`int`	`0`
`errored`	`bool`	`False`
`error`	`str`	`''`
`origin`	`str`	`SOURCE_NONE`

Properties

property has_output: bool #[source]: No description.

property has_answer: bool #[source]

Output that states something, as opposed to a traceback.

The distinction decides the merge below. When the grading run cannot find the data, every cell after the first read_file raises, and a raise is output — so ranking cells by "has output" hands every slot to the failed run and hides the answers the student actually saved.

class NotebookAnswers #[source]

dataclass

Every answer a notebook shows, with where each one was seen.

Fields

Name	Type	Default	Notes
`cells`	`list[CellAnswers]`	`field(default_factory=list)`
`source`	`str`	`SOURCE_NONE`

Properties

property figures: int #[source]: No description.

property error_cells: list[CellAnswers] #[source]: No description.

property cells_with_output: int #[source]: No description.

property cells_with_answers: int #[source]: No description.

Methods

shows(expected: float, rel_tol: float = 0.0, abs_tol: float = 0.0) -> CellAnswers | None #[source]: The first cell that printed or returned this number.

shows_all(expected: Iterable[float], rel_tol: float = 0.0, abs_tol: float = 0.0) -> bool #[source]: No description.

shows_together(expected: Iterable[float], rel_tol: float = 0.0, abs_tol: float = 0.0) -> CellAnswers | None #[source]

A single cell that shows all of these numbers at once.

"10" on its own means nothing — it is in every notebook. "10 and 32618 in the same output" is a ten-band scene in UTM zone 18N, which is the file this assignment ships.

shows_rows(expected: int, tolerance: int = 0) -> CellAnswers | None #[source]: A frame of this many rows, from a repr footer, .shape or .info().

frames_with_columns(names: Iterable[str]) -> list[CellAnswers] #[source]: No description.

describes(key: str, expected: float, abs_tol: float) -> CellAnswers | None #[source]: No description.

charts() -> list[HoloviewsRepr] #[source]: No description.

note() -> dict[str, Any] #[source]: No description.

Functions

read_answers(path: str | Path | None, origin: str) -> NotebookAnswers #[source]: Harvest one notebook's saved outputs.

collect_answers(executed_path: str | Path | None, submitted_path: str | Path | None) -> NotebookAnswers #[source]

Answers from both notebooks, merged cell by cell.

Two notebooks may show the same answer: the one this grading run executed, and the one the student handed in with its outputs already saved. Neither is reliably the better source — the executed run can die on a data path the grader could not resolve, and a saved output can be stale or hand-edited — so they are merged rather than ranked, and each answer keeps the note of where it came from.