📉 The premise: nobody reads the code 0:00
The graph he opens with is real: Anthropic's own engineering post reports about 2,500 changes merged per day in May 2026 and over 30,000 per day by September. His conclusion — almost nobody at Anthropic or OpenAI reads every line of code any more — is not a dig; it is Anthropic's own description of their process. Their engineers, in the post, "review every change request for the system, but they now examine the intent of the change as opposed to reading every line of code."
So the question the video sets out to answer is the right one: if reading is out, what replaced it? His answer is verifiers — and the rest of the talk is the most concrete treatment of the idea I have seen from a practitioner.
🔀 The Anthropic example: shadow traffic 0:36
His anchor case comes from a podcast interview he cites (Latent Space's "How Anthropic Ships"): Anthropic rewrote the Messages API from Python to Rust, and at the start nobody on the team was a Rust developer. Claude wrote the code. Before any of it existed, the team stopped and asked: what is the verifier here — how will we know the new version is correct?
The answer was shadow traffic: run the new Rust service alongside the old Python one, send both the same real requests, and compare responses. Any divergence becomes a bug report for Claude, which fixes it, and the loop repeats until the answers match — then traffic migrates gradually. Reported motivation: a compute constraint, with CPU usage on the API dropping 40–50%.
🧩 What a verifier is: surface, answer key, comparison 2:44
His definition is the most useful sentence in the video: a verifier is a check your agent cannot argue against. Every good one has three parts.
| Part | What it is | Examples |
|---|---|---|
| Surface | The same interface a real user or system meets the change through | Web page, API, MCP server, CLI, mobile or desktop app, a phone call for voice agents |
| Answer key | What "correct" looks like — and critically, the agent did not write it | An old system doing the same job, a snapshot, an independent test suite, a spec |
| Comparison | A mechanical check of whether what happened matches the key | Response diff, byte equality, pixel diff, pass/fail of an e2e flow |
🚫 Why agent-written tests fail 2:12
His critique of the default workflow is blunt and, in my experience, correct: when an agent writes unit and integration tests, it is mostly restating what the code already does. All the tests go green, and the button still does nothing when you open the page.
The failure mode is the response to it. Engineers add more rules to their CLAUDE.md — "always write proper tests", "never weaken an assertion" — hoping better instructions produce better tests. He argues the frame is wrong: the answer is not better test-writing, it is verification systems, tooling and environments.
🗝 The answer key hierarchy 3:40
Not all answer keys are equal. Ranked by strength, from his talk:
| Answer key | Strength | Why |
|---|---|---|
| An existing old system doing the same job | Strongest | Real behaviour on real inputs — that is what shadow traffic exploits |
| A snapshot / byte-for-byte artefact | Strong | Deterministic, unaffected by opinion; a diff either exists or does not |
| An independent test suite not derived from this code | Strong | Playwright-style end-to-end tests meet the change at the user surface |
| A specification's expected-behaviour table | Weaker | Depends on an agent judging whether the expectation was met |
| No answer key | Worst | The agent decides for itself what "verified" means — green tests, broken buttons |
His split for choosing: is this a refactor or a new feature? A refactor has an old system or a snapshot available, so use it. A new feature has no prior behaviour, so the spec becomes the key and agentic end-to-end tests become the comparison.
📋 Spec as answer key: the table at the bottom 4:07
The concrete technique is small and immediately usable: end every spec with a table of user action → expected result. For a tagging feature:
| User action | Expected result |
|---|---|
| A user adds a tag to a note | The tag shows on the note immediately |
| A user filters notes by tag | Only notes with that tag are shown |
| A user deletes a tag entirely | The tag is removed from every note and from the tag list |
Ten to fifteen rows, depending on complexity, and that table is the answer key. After implementation, each row becomes an end-to-end test — which is how he keeps the key from expiring the moment the commit lands.
🤖 Agentic end-to-end: e2e and the judge problem 5:04
He recommends e2e, an open-source TypeScript testing framework for web and mobile that mixes deterministic assertions with agentic steps. In his example: a member upgrades to Pro → open the billing page → an agent with browser use actually performs the upgrade → a second agent on a different, cheaper model judges "the invoice preview shows a prorated amount".
| Property | Why it matters |
|---|---|
| Actor and judge are different agents, often different models | Removes the self-grading problem that ruins agent-written unit tests |
| Deterministic steps where possible, agentic steps only where needed | You keep pass/fail rigour without scripting every pixel |
| Caching, so you pay for tokens mainly when a test fails | Makes agentic verification affordable at suite scale |
| Runs on the real surface (browser, app) | Catches the "green tests, dead button" class directly |
🗑 Deleting the test suite in favour of E2E 5:53
The boldest claim in the video, and the one to handle carefully: for some of his own projects he deleted the tests — one pull request removing 110,000 lines, another 295,000 lines — replacing them with end-to-end tests, and says he now catches more bugs per PR.
🧬 Byte-for-byte: decompilations and terminal protocols 11:40
The cleanest verifier of all is equality of bytes, and he finds two live examples:
- Matching decompilation of old games. Agents rebuild source from a binary, and the check is that recompiling with the same vintage compiler reproduces the exact original machine code. He points at a faithful rewrite of Crimsonland (2003) produced by agents running for hundreds of hours. Here the answer key is the original game and the comparison is bytes: no judgement, no rubric, no drift.
- Mitchell Hashimoto making Ghostty display images the way Kitty does. The answer key is Kitty's behaviour, because programs already target it; the verifier is whether the same programs render the same images in the new terminal.
🥟 Bun in Rust: the test suite as answer key 11:09
His third example is the most directly copyable: Bun rewrote its runtime from Zig to Rust, and the answer key was Bun's existing test suite — which happens to be written in TypeScript, so it does not depend on the runtime's own implementation language. Claude and subagents worked the migration, hill-climbing until the suite passed again.
Checked against Jarred Sumner's own write-up, the account holds and gets sharper:
| Detail | Record |
|---|---|
| Rewrite from Zig to Rust | Verified — "Rewriting Bun in Rust", 8 July 2026 |
| Claude wrote most of it | Verified — Sumner used a pre-release Claude Fable 5 for much of the rewrite |
| Test suite in TypeScript as the answer key | Verified — and it is why the key survived a language change |
| Goal was 100% passing with nothing skipped | Verified — and Sumner adds the step Ray's summary omits: he manually verified the tests were actually running and not being skipped |
| Corporate context | Bun was acquired by Anthropic in December 2025; the rewrite is an in-house Anthropic project |
🖼 Pixel diffs as verifiers 12:49
A screenshot is an answer key too. He demonstrates a builder-verifier loop that copies a website's design with a pixel diff as the check, and the convergence curve is the illustration: 19% → 1.8% → 0.54% → 0.52%, stopping there, dropping to 0.07% if you remove the logos.
Two honest readings of that curve. First, the last stretch is where the remaining difference is mostly logos and assets you have no right to copy anyway. Second, the plateau is informative: a verifier with a floor of ~0.5% tells you the loop has extracted what it can, and further iterations would be chasing noise — which is exactly the signal you want a verifier to give you.
He notes the same method applies to making font files smaller for faster page loads, where the pixel diff confirms the text still renders identically.
🔁 Builder ↔ verifier, and one real failure 9:58
The operating pattern is a two-agent loop: the builder implements, the verifier proves, and any failure goes back to the builder until the check passes. He runs most of it from Slack sandboxes rather than a terminal, and shows a recording where a desktop app's new microphone picker was verified across user flows — with the verdict "on a real Windows machine, it passed four of the five flows", which sent it back for another round.
🧰 Maintaining verifiers after an escape 15:26
His most instructive war story is about a verifier's blind spot rather than a bug. A video player change passed his checks and broke on Safari — because the verifier ran Playwright's Chromium only, and Safari and Chrome handle video playback differently. A user reported it.
The fix is the lesson: rather than patching the change, he patched the verifier. Anything touching video playback must now be checked with Playwright's WebKit build and on a real Mac the agent has access to. He calls the tooling "prove it" skills, and the general rule is: when a bug escapes, improve the verifier upstream, not the prompt.
🧪 Auditing verifiers with mutation testing 17:31
The third job, and the one most teams never do: prove the verifier still catches things. His method is mutation testing with an agent — take a recent change, plant a realistic bug (an off-by-one in a counter), then hand the flow to a fresh agent that is told to verify it and not told what broke. If the verifier catches it, the check is alive. If it does not, the verifier needs fixing.
| Job | Question it answers | Failure if skipped |
|---|---|---|
| Create | What is the surface, and what is the answer key? | No verifier at all; the agent grades itself |
| Maintain | What did users hit that we did not check? | The verifier silently certifies a class of bug |
| Audit | Would this verifier still catch a planted bug? | A dead or weakened check that looks green forever |
His own skills implement this trio as create-prove-it, maintain-prove-it and audit-prove-it (MIT-licensed, small repo, actively updated) — and the create step is itself generative: it maps the surfaces, checks they are reachable, finds candidate answer keys, discusses them with you with a cost estimate per method, and writes a reusable verification skill for that surface.
✅ Claims checked
| Claim | Verdict | Evidence |
|---|---|---|
| Anthropic merges tens of thousands of changes a day, and engineers review intent rather than every line | Verified | Anthropic engineering post, "Redesigning source control at Anthropic" (13 Oct 2026): ~2,500/day in May, 30,000+/day by September; "they now examine the intent of the change as opposed to reading every line of code" |
| Anthropic rewrote the Messages API from Python to Rust; nobody on the team was a Rust developer; shadow traffic as the verifier; 40–50% CPU reduction | Relayed from a podcast | Attributed to the Latent Space interview he cites; not in Anthropic's written post, so unverified here |
| Agent-written tests restate the implementation, so they pass while the feature is broken | Sound and widely observed | The mechanism — the agent writes the answer key too — is the core of his argument |
| A verifier needs a surface, an answer key the agent did not write, and a comparison | Coherent framework | Not a standard definition, but internally consistent and operational |
| e2e is an open-source AI testing framework mixing deterministic and agentic steps, with a separate judge agent and caching | Verified | e2e.tester.army — "The open source AI testing framework", TypeScript, npx e2e@latest init |
| He deleted 110,000 and 295,000 lines of tests, replacing them with e2e, and catches more bugs per PR | Self-reported | His own PRs; no measurement offered, and the risk profile of the project is unstated |
| Crimsonland was rebuilt as a matching decompilation by agents over hundreds of hours | Verified existence | banteg/crimson — "a faithful rewrite of crimsonland (2003)", Python, ~195 stars, no licence file |
| Ghostty added Kitty-compatible image display, with Kitty as the answer key | Tweet-attributed | Consistent with Ghostty's 1.4 release notes (full Kitty graphics); the specific tweet was not opened here |
| Bun rewrote its runtime from Zig to Rust using Claude, with its TypeScript test suite as the answer key, hill-climbing to 100% passing with nothing skipped or deleted | Verified | "Rewriting Bun in Rust" (8 Jul 2026): Claude Fable 5 pre-release used for much of the rewrite; "Once 100% of Bun's test suite passed in CI on all platforms (and I manually verified the tests were in fact running and not being skipped)" |
| Bun is independent of Anthropic | Not stated, but relevant | Bun was acquired by Anthropic in December 2025 — so this is Anthropic's own case study, not a third-party |
| Pixel diff converged 19% → 1.8% → 0.54% → 0.52%, and 0.07% without logos | Demonstrated, his machine | Shown on screen; a screenshot key requires a stable render environment to be meaningful |
| His skills are free and open source | Verified | ray-amjad/skills — MIT, "Verification skills for coding agents", updated the day of the video |
| The Safari video-playback bug escaped because the verifier only ran Chromium | Plausible and instructive | A textbook example of verifier blind spots; his fix (WebKit + real Mac) is the right shape |
| Mutation testing by planting a bug and asking a fresh agent to find it | Correct technique | Standard mutation testing, applied to agentic verifiers — the fresh-context requirement is the important detail |
Transcript notes. Auto-captions render "cloud code" (Claude Code), "Enthropic" (Anthropic), "cloud md" (CLAUDE.md), "prove it"/"private skill" (his "prove-it" skills), "E2 library"/"E2E" (the e2e framework), "bite for bite" (byte-for-byte), "playright" (Playwright), "AMCP server" (MCP server), "Jeb" (a cheap model name, unclear), "Crimson Land" (Crimsonland), "Ghosty" (Ghostty), and "Go to Rust"/"Go to rust" in his hypothetical migration example.
🔑 Key takeaways
- Ask "what is the verifier here?" before the agent writes anything. The question, asked early, is what turns an open-ended change into a measurable one.
- A verifier is a check the agent cannot argue with: a real surface, an answer key the agent did not write, and a mechanical comparison.
- Agent-written tests are self-graded exams. The same agent writes the answer key, so green is guaranteed and meaningless.
- The strongest answer key is an old system doing the same job — which is why shadow traffic beats any test suite you could write.
- Rank your keys: existing system > snapshot/byte equality > independent e2e > spec table > nothing. Then pick by refactor versus new feature.
- A spec should end with a user-action/expected-result table. Ten to fifteen rows become the answer key and then become e2e tests after implementation.
- Separate the actor from the judge. The agent performing the flow should not be the agent deciding whether it worked — ideally a different, cheaper model.
- Byte equality is the cleanest verifier there is — matching decompilations and Kitty-compatible rendering both inherit an oracle rather than inventing one.
- Bun's rewrite worked because the test suite was in TypeScript, so the key survived a change of implementation language. Keys that depend on the code they check do not survive rewrites.
- Audit the verifier, not just the code: Bun's author manually confirmed the tests were actually running and not skipped — that is the step that makes "100% passing" mean anything.
- When a bug escapes, fix the verifier upstream, not the prompt. A Chromium-only check silently certifies every change as cross-browser-safe.
- Mutation-test your verifiers: plant a realistic bug, hand the flow to a fresh agent, and see if it notices without being told what broke.
- Choosing a verifier is a budget decision. Shadow traffic gives the best coverage and costs the most to run — the strongest keys are the ones you pay to keep alive.
🔗 Sources
- The source video — Ray Amjad, "TOP 1% Agentic Engineering: Claude Code Verifiers", 19:15, 1 November 2026
- Anthropic: Redesigning source control at Anthropic (13 October 2026) — the merge-rate chart, Scion, and "review intent, not every line"
- Latent Space: How Anthropic Ships — the interview he cites for the Python→Rust Messages API rewrite and shadow traffic
- e2e — the open-source AI testing framework (TypeScript, browser/mobile, deterministic + agentic steps)
- ray-amjad/skills — his create/maintain/audit "prove-it" verification skills (MIT)
- Bun: Rewriting Bun in Rust (Jarred Sumner, 8 July 2026) — the test suite as answer key, and the manual check that the tests really ran
- banteg/crimson — a faithful rewrite of Crimsonland (2003) used as a matching-decompilation example
- Mitchell Hashimoto on Ghostty and Kitty's image protocol — byte-compatible rendering as the answer key
- The Code Nobody Reads — the essay he cites alongside the video