Selected Work - AI

BUC Populi - shipping an LLM grader faculty actually trust

An AI grading engine on Anthropic Claude: rubric-anchored, per-criterion essay scoring for professors. The hard part wasn't model accuracy - it was trust. Every AI result is a draft a human approves, and the whole thing is built so quality is provable, not hoped for.

GRADING ENGINE· claude-sonnet-4
live
essay_4821.docx
Thesis
18/20
Evidence
16/20
Structure
20/20
Grammar
17/20
86/100
confidence: high
no plagiarismon-topiccitations ✓

DRAFT · pending_review

awaiting human approval · never auto-pushed

Role

Solo Engineer / AI

Focus

Prompting · Human-in-the-loop · Trust

Links

Internal tool

01The AI problem

Grading written work is the kind of judgment task an LLM can do well, and the kind of high-stakes output you cannot let a model own. So the question was never whether the model can grade, it was how to ship an AI grade a professor will stake their name on. The answer was one model and plain rules behind one human gate: Claude scores each essay against a structured JSON rubric, and a rules-based audit flags the attendance cases a person should check. Both produce drafts a human approves, so the AI carries the volume and the human keeps authority.

02What I built

  • Rubric-anchored grading prompt - a structured prompt built from the institution's JSON rubric, sent to Claude, scoring each criterion with justification, so output is structured, consistent, and reviewable rather than a freeform guess.

  • Rules-based attendance audit - a decision matrix over attendance and submission signals - online and in-person weeks, late work - flagging the exact cases a human should check. Plain code, because a rule does this job better than a model.

  • Drafts, never auto-submits - every model output is held at pending_review; nothing is written to a student's record without a human approving it. The gate is the core of the system, not a setting.

  • Built to be checked - structured outputs, per-criterion scores, and surfaced confidence so results can be reviewed at a glance and spot-checked against the rubric - quality you can verify, not take on faith.

03AI engineering decisions

Approval gate before any grade

Problem
An AI grade flowing straight into a record is unacceptable academically and ethically, and no benchmark score makes faculty comfortable with autonomous grading.
Approach
I made human approval the product. The AI drafts, a professor approves, and nothing auto-submits, so the model's job is a strong, reviewable first pass.
Result
Faculty who would have rejected AI grading on day one actually use it, because they stay the system of record.

A model for judgment, rules for the rest

Problem
A generic grade-this-essay prompt gives inconsistent output, and the attendance audit needs to catch a specific pattern rather than summarize text.
Approach
I anchored the grading prompt to the institution's rubrics for consistent per-criterion scoring, and wrote the attendance audit as plain rules instead of asking a model.
Result
Consistent grade drafts and a predictable audit flag a human can verify quickly, instead of one black box.

Built for fast review

Problem
If reviewing the AI's output is as slow as doing the work, the AI has not actually saved anything.
Approach
I surfaced the flags and structured per-criterion results so a professor reviews and corrects rather than re-grading from scratch.
Result
Grading time drops sharply while the professor keeps full control of the decision.

Strict JSON from the model

Problem
An LLM that returns prose where you expect per-criterion scores breaks the pipeline and the review UI, and models like to add a chatty preamble.
Approach
The grading prompt requires strict JSON for scores, feedback, flags, and confidence, parsed with a fallback that extracts the object if it is wrapped in text, and empty or too-short submissions short-circuit before reaching the model.
Result
Every result is structured data a reviewer can scan in seconds rather than free text to interpret.

04How it was built

Model-first and small, with the LMS as the source of truth under a human gate.

Anthropic ClaudeStructured JSON outputsRules-based auditPython 3.12Populi API v2Streamlit

05Outcomes

Claude Sonnet 4

Grader prompted with a JSON rubric

Human-approved

No AI grade ever auto-pushed