Run one cohort of real student submissions through competing assessment methods — pure-LLM marking, deterministic signals, and a hybrid — then see which one is repeatable and which agrees with your marks.
The bench measures; it never marks students.
Free & open source (MIT). No account. Results stay on your machine.
# prefer the command line? Same engine, on PyPI:
pip install assessment-benchEvery experiment compares “arms” on the same rubric and the same cohort, with repeated runs so inconsistency has nowhere to hide.
An LLM reads each submission against your rubric and emits a mark — the approach everyone is tempted by, kept honest as the baseline under test.
Deterministic evidence from the lens analyser family via assessment-lens — the same submission always yields the same observations. No LLM involved.
LLM marking with the deterministic signals included in the prompt — does grounding the model in measurements make it more trustworthy?
Point the app at your materials.
A rubric, a folder of submissions (one subfolder per student), and — optionally — a CSV of your own marks to enable agreement statistics.
Pick the methods to compare.
Add arms in a form — no config files. Choose a provider per arm, paste an API key once, set how many repeated runs you want.
Read the findings in plain language.
“LLM marking agreed with your marks at r=0.82 but varied ±3 points between runs.” Full statistics (Pearson, Spearman, reliability) sit one scroll below, and every run lands on disk as JSON + CSV for your own analysis.
First launch installs the analysis engine automatically (one time, with progress shown). After that it works offline with local models.
The engine runs locally. Choose Ollama for the LLM arms and student work and marks never leave your machine — no cloud, no vendor, no consent-form headaches for your ethics application.
Compare models against each other — each arm picks its own provider.
The desktop app and the PyPI
package are the same product. Use the app for point-and-click experiments;
use assessment-bench run experiment.yaml when you want scripted,
reproducible pipelines — results are identical.