Rubric Anatomy

What a rubric actually is, the forms it takes in real files, and the design options you choose when you write one. A rubric turns an LLM-as-judge from a vibe into a verifier.

A rubric is only trustworthy when it's specific enough that "two domain experts would independently reach the same pass/fail verdict." — Anthropic, Demystifying evals for AI agents

It's a prompt artifact, not a config format

A rubric is the written grading criteria handed to a judge so a verdict is reproducible. The medium is usually plain markdown or a structured list, living either inline in the judge prompt or in its own versioned file. It has four parts:

Form 1 — Inline checklist in the judge prompt most common

# Grade the support agent's reply. For EACH criterion: PASS or FAIL.
# If you lack the information to decide, answer UNKNOWN.

- empathy:    The agent acknowledged the customer's frustration.
- grounded:   Every claim is grounded in the fetch_policy tool results.
- resolution: The refund decision is stated explicitly and clearly.
- tone:       No defensive or blaming language.

Output JSON: {"empathy":"PASS|FAIL|UNKNOWN", ..., "overall":"PASS iff all PASS"}

Those assertion lines are the rubric. Note: binary verdicts · an UNKNOWN escape hatch · all-must-pass aggregation. (Anthropic)

Form 2 — A standalone rubric file the grader points at

Anthropic's coding-agent grader references a separate prompts/code_quality.md — rubrics are files, versioned in git next to the code they grade.

# Code-change rubric — grade the DIFF only.

## Correctness (all must PASS)
- Implements every item in the task's acceptance criteria.
- No existing test was weakened, skipped, or deleted to go green.
- Handles the empty / null / boundary input named in the ticket.

## Hygiene (all must PASS)
- No secrets, debug prints, or commented-out code left behind.

## Verdict
Return PASS only if every item passes. Else FAIL and list each failing
criterion with the file:line that caused it.

Form 3 — Few-shot critique examples critique shadowing

Hamel Husain's method: a domain expert writes a binary verdict plus a detailed critique for real cases; you paste them in as few-shot anchors. The critique must be detailed enough that "a new employee could understand it."

<example-1>
  <nlq>show me traces where ip is 10.0.2.90</nlq>
  <query>{ "filter":{"field":"ip","op":"=","value":"10.0.2.90"} }</query>
  <critique>{
    "critique": "Correctly filters on ip with the right operator. No
                time range, but the default is acceptable here. It passes.",
    "outcome": "good"
  }</critique>
</example-1>

Then measure agreement between judge and expert; refine until aligned. Husain reached >90% agreement in three iterations — that number is what makes the rubric trustworthy.

Form 4 — In Claude Code: the rubric IS the verification subagent

The most useful medium for this mission. YAML frontmatter is plumbing; the markdown body is the rubric. Fresh context, sees only the diff, grades, returns a verdict.

---
name: diff-grader
description: Grades the current diff against the rubric. Returns PASS/FAIL.
tools: Read, Bash
---
You are an adversarial reviewer. You did NOT write this code.
Review ONLY the diff. Grade each criterion PASS/FAIL (UNKNOWN if unsure):

1. Every acceptance criterion in the linked issue is implemented.
2. No test was deleted or weakened to pass.
3. Error paths return actionable messages, not bare codes.

Report ONLY correctness/requirement gaps — no stylistic refactors.
End with one line:  VERDICT: PASS   or   VERDICT: FAIL — <reasons>

The "only correctness gaps" bound matters: a reviewer told to find problems "will usually report some, even when the work is sound." (Sub-agents) The rubric defines what counts as failure.

Sibling media for the same rubric: a /goal condition (re-checked each turn), a .claude/commands/grade.md slash command, or an acceptance-criteria block in a spec file / CLAUDE.md.

The options menu — what you choose when you write one