Data-Viz Assignment 2 — 24 submissions

3 arms · 5 repeated runs per LLM arm · marks out of 100 · compared against the lecturer's marks

The bench measures how each method agrees with the human marks. It never marks students.

What this run found

“hybrid-claude” showed strong agreement with your marks (r=0.87 across 24 submissions) and varied ±2.3 points between repeated runs.
“llm-claude” showed strong agreement with your marks (r=0.81 across 24 submissions) and varied ±5.6 points between repeated runs.
“llm-local” (Ollama, llama3.2) showed moderate agreement with your marks (r=0.58 across 24 submissions) and varied ±11.2 points between repeated runs.
The single signal that best tracked your marks was document.word_count (r=0.71) — moderate agreement without any LLM.
Submissions typical of the cohort tended to receive higher marks (similarity vs marks r=0.44) — worth a look at how the distinctive ones were marked.

Agreement with human marks

measure	n	Pearson	Spearman
hybrid-claude arm	24	0.874	0.851
llm-claude arm	24	0.812	0.790
document.word_count signal	24	0.708	0.731
code.cell_count signal	24	0.612	0.640
llm-local arm	24	0.583	0.529
distinctiveness.mean_similarity	24	0.442	0.418
document.reading_grade signal	24	0.187	0.203

Arm repeatability

arm	reliability	between-run spread
hybrid-claude	0.96	scores varied ±2.3 points
llm-claude	0.91	scores varied ±5.6 points
llm-local	0.82	scores varied ±11.2 points
signals	—	deterministic (identical every run)

Everything above also lands on disk as result.json, runs.csv, signals.csv, distinctiveness.csv and agreement.csv for your own analysis. Reliability is 1 − coefficient of variation over repeated runs; agreement is Pearson/Spearman against the marks you supplied.