Lesson 6 · Agentic Loop Engineering

Calibrating the Judge

Lesson 2 ended with a warning: an unmeasured rubric is a vibe. This is the measuring — and you'll do it yourself, by being the judge that gets graded.

⏱ ~10 min🎯 The agreement number🔗 Completes the rubric arc (L2→L3)

1 · The trust problem, stated plainly

Your diff-grader works because most of its criteria bottom out in something runnable. But the moment a rubric criterion requires judgment — "the error message is actionable," "the summary kept the key facts" — your verifier is a model giving opinions. Anthropic's standard: LLM-as-judge graders "should be closely calibrated with human experts to gain confidence that there is little divergence between the human grading and model grading." (Demystifying evals)

The trustworthiness of a judge is not an adjective. It's a number: the measured agreement rate between the judge's verdicts and yours, on examples you graded independently. — the operating principle behind Husain's critique shadowing (he reached >90% agreement in three iterations)

2 · The procedure: critique shadowing

  1. Collect 10–30 real examples of the thing being judged (not synthetic toys — the weird ones are the valuable ones).
  2. You (the domain expert) grade each one: binary verdict + a written critique detailed enough that "a new employee could understand it." Terse critiques are the common mistake.
  3. Build the judge prompt: rubric + a handful of your verdict-and-critique pairs as few-shot anchors.
  4. Run the judge on the same examples. Count agreements. A spreadsheet is enough.
  5. Study the disagreements — they are gold. Each one is either a judge error (add a clarifying example) or your rubric being genuinely ambiguous (tighten the criterion).
  6. Iterate until agreement is high enough for the stakes. Then spot-check periodically — domains drift.
The reframe most people miss: calibration improves your rubric as much as the judge. Husain notes that reviewing the LLM's critiques helped the domain expert articulate their own criteria more clearly. Disagreement analysis is rubric development.

3 · The lab — you be the judge

Here's the experience from the other side. Below is a rubric and six real-ish CLI error messages. Grade each PASS/FAIL. Then reveal the "expert" verdicts and see your agreement rate — and notice which disagreements expose rubric ambiguity versus your own miscalibration.

Rubric — an error message PASSES only if all three hold:
① names what failed  ·  ② names the offending input/value  ·  ③ says what to do next

4 · Reading your number

6/6 — you and the expert share a rubric interpretation; this rubric is ready to hand to a model. 4–5/6 — normal first pass; the disagreements tell you which criterion is ambiguous (item 5 is the designed trap: criterion ② is genuinely arguable for it, which means the rubric needs tightening, not the judge). ≤3/6 — the rubric isn't carrying the meaning you think it is; rewrite before automating. Now substitute "LLM" for "you": that's exactly the measurement loop you run before trusting a judge in your verification loop — and the same disagreement analysis drives the same rubric fixes.

Operational defaults, all source-backed: binary verdicts (scales don't correlate with expert judgment), critique-then-verdict ordering, an "Unknown" escape hatch (exclude those from the rate, but investigate them), and recalibrate when the domain shifts.

💬 Make it real: pick something from your actual work — PR review comments, API errors, commit messages — and I'll run real critique shadowing with you: you grade ~10 of them in chat, I'll draft the rubric and judge prompt, we measure agreement, and iterate. That's the full Lesson 6 loop on live data.

Where to go next