The Feature-List Ledger

Ground truth for work too big for one context window — Anthropic's antidote to agents declaring victory at 30%. Plus the twist: a ledger is only ground truth if it can't lie.

1 · Two failure modes of long-running work

When Anthropic built a harness to have agents construct a full claude.ai clone over many sessions, two failure modes dominated — both verbatim from the writeup:

Notice both are verify-step failures at the scale of sessions. Within one turn, you learned to gate "done" on a check. But a fresh session "begins with no memory of what came before" — so what gates "done" across fifty sessions? The answer can't live in the context window, because the context window dies. It has to live in the environment.

2 · The pattern: state in files, ground truth in a ledger

Anthropic's harness puts all cross-session state in committed files: a claude-progress.txt log, an init.sh to start the dev server, git history — and the centerpiece, feature_list.json: 200+ granular features, every one initially "passes": false, created once by an initializer agent, then touched by every later coding agent in exactly one way:

{
  "category": "functional",
  "description": "New chat button creates a fresh conversation",
  "steps": ["Navigate to main interface", "Click the 'New Chat' button", …],
  "passes": false
}

The session protocol, every session: orient (read git log + progress files) → choose "the highest-priority feature that's not yet done" → work on one feature at a time → verify end-to-end with browser automation, "testing as a human user would" → flip passes → commit. And the rules around the ledger are deliberately blunt:

"It is unacceptable to remove or edit tests because this could lead to missing or buggy functionality." — agents may "edit this file only by changing the status of a passes field." — Anthropic, Effective harnesses for long-running agents

3 · The files — demo scale

Demo 03 shrinks the pattern to a string-utility library: 6 features, 2 done. For a web app each feature's verifier is browser steps; for a library it's a test pattern — same pattern, different rung of the taxonomy.

{
  "id": "truncate",
  "description": "truncate(str, max): cut to at most max chars, ending with … when truncated",
  "test_pattern": "truncate",
  "passes": false
}

1. Orient: read claude-progress.txt, git log --oneline -10, features.json.
2. Choose the FIRST feature with "passes": false. ONE feature per session.
3. Implement it in src/strkit.js.
4. Verify: node --test --test-name-pattern "<id>". Flip "passes" to true
   ONLY after you have seen that command pass.
5. Append a dated line to claude-progress.txt.  6. Commit, then stop.

## Hard rules
- Edit features.json ONLY by changing a "passes" value.
- It is unacceptable to remove or edit tests — they are the specification.
- A red overall suite is expected mid-project.

Note that last line — the contrast with Lesson 4. There, the gate was "whole suite green." Here the suite is supposed to be red mid-project; the gate is different: the ledger must not lie.

4 · The gaming problem — and the gate that closes it

Here's the flaw in a naive ledger: "passes": true is just a bit. Nothing physically stops an agent (or a careless human) from flipping it without doing the work — and then every future session inherits a corrupted ground truth. Anthropic leans on strongly-worded instructions; words, as you know from Lesson 4, are advisory. So demo 03 composes the two lessons: a Stop hook that re-runs the verifier for every "passes": true claim before any turn may end.

for (const f of features.filter((f) => f.passes)) {
  const r = spawnSync('node', ['--test', `--test-name-pattern=${f.test_pattern}`], …);
  // count ✔ lines whose test NAME matches — node reports the test FILE as
  // one passing test even when the pattern matches nothing, so the summary
  // counts alone can't be trusted
  const matched = out.split('\n')
    .filter((l) => l.includes('✔') && new RegExp(f.test_pattern).test(l)).length;
  if (r.status !== 0 || matched === 0) lies.push(…);
}

That comment is a battle scar, not decoration: the first version of this checker trusted node's summary counts and would have approved a feature whose test_pattern matched zero tests — a verifier that silently passes is worse than no verifier. Caught only by testing the checker against a deliberately corrupted ledger. Verify your verifiers.

5 · The dialogue — one session, then the cheat

And the cheat — try this yourself in the demo:

LEDGER MISMATCH: features.json claims do not match reality.
- "truncate" claims passes:true but its verifier fails
- "camelCase" claims passes:true but its verifier fails
- "initials" claims passes:true but its verifier fails
- "wordCount" claims passes:true but its verifier fails
Fix the implementation or set the claim back to false. Do not edit tests.

6 · Check yourself

Where to go next