Designing a Rubric

The LLM-as-judge is only as good as the rubric you hand it. Here's how to write one that turns a vibe into a verifier — and a hands-on clinic to practice.

1 · Why the rubric is the whole game

In Lesson 1, the LLM-as-judge was the verifier you reach for when there's no deterministic check — "is this error message clear?", "did the summary keep the key facts?". But a judge with a vague instruction ("rate the quality 1–10") is just a second opinion with no spine. The rubric is what gives it ground truth.

A rubric must be specific enough that "two domain experts would independently reach the same pass/fail verdict." — Anthropic, Demystifying evals

If two careful humans reading your rubric would disagree, the model will too — and inconsistently. That's the test for every criterion you write.

2 · A rubric is a prompt artifact — here's the medium

It's not a config schema. It's markdown criteria, living either inline in the judge prompt or in its own versioned file. The canonical Claude Code medium is a verification subagent whose body is the rubric:

# .claude/agents/diff-grader.md  — the markdown body IS the rubric
You did NOT write this code. Review ONLY the diff. Grade each PASS/FAIL
(answer UNKNOWN if you can't tell):
1. Every acceptance criterion in the linked issue is implemented.
2. No test was deleted or weakened to pass.
3. Error paths return actionable messages, not bare codes.
End with one line:  VERDICT: PASS  or  VERDICT: FAIL — <reasons>

Four forms in total (inline checklist · standalone file · few-shot critiques · subagent) are laid out in your reference → Rubric Anatomy ↗

3 · The five design choices

Binary. "People don't know what to do with a 3 or a 4," and scale scores don't correlate with expert judgment. Force PASS/FAIL. (Husain)
Decomposed. One criterion per dimension, each graded by its own isolated judge — not one blob grading everything. (Anthropic)
Critique first. Make the judge write its reasoning before the verdict, so the verdict is earned, not guessed. (Husain — "critique shadowing")
Calibrate. Seed the prompt with expert-written critiques, then measure judge-vs-human agreement and refine. Husain hit >90% in three rounds. That number is your trust.
Escape hatch. Let the judge answer "Unknown" when it lacks the info, so it doesn't fabricate a verdict. (Anthropic)

4 · Weak → strong, in one example

# WEAK — unverifiable, scaled, holistic
Rate the summary's overall quality from 1 to 10.

# STRONG — binary, decomposed, checkable, with an out
For EACH, answer PASS / FAIL / UNKNOWN:
- coverage:  Every numeric figure in the source appears in the summary.
- faithful:  No claim in the summary is absent from the source.
- length:    The summary is 3 sentences or fewer.
overall: PASS iff coverage AND faithful AND length all PASS.

5 · Clinic — be the rubric doctor

Five weak rubric criteria walk into your loop. For each, pick the fix that makes it a real verifier. Immediate feedback + rationale.

Where to go next

Suggested Lesson 3 (ask me to build it): calibration in practice — run a rubric against 10 examples, compare to your own verdicts, and compute the agreement number that makes it trustworthy.