0:00The row that stood out in Google Research's Harness² paper: GLM5 goes from 64.2 to 75.7 on the WorkBuddy benchmark with its weights frozen — passing Gemini 3.5 Flash's untouched 72.4 in the same OpenCode agent runner. If your agent flops on a task its harness wasn't built for, this is the fix you'd otherwise pay a bigger model (or manual harness tuning) for. Cloud Codes reads the actual method section to explain what one step does, what it costs, and who should bother.
The headline claim — and its caveat
0:47The popular take, from a Substack post titled "The Harness Beats the Model", is that the harness matters more than the model — partly right, because every gain here comes from editing the system around a frozen model. But the nuance: Gemini 3.5 Flash with the same treatment still finishes ahead at 77.5. So it's not simply "a smarter harness beats a stronger model" — it's narrower. (And Google DeepMind's Logan Kilpatrick says models will have "digested a good chunk of this scaffolding within about a year" — more below.)
What the agent is allowed to touch
1:28The README lists eight pieces: system prompt, guardrails, memory, skills (what the model is told and remembers), tool guidance and tool scripts (how it uses tools), subagents (handing work around), and plugins (middleware). Model weights are not on the list. The reframe: a person normally designs the harness once and every new task runs on it; here the harness itself is the thing that gets improved.
One step, four phases
1:571. Reference run. The agent attempts the task with its current harness; its trajectory (the steps taken) and deliverable (what it handed back) become the baseline.
2:212. Propose. An editor model call — given the task, the current harness, the reference run, earlier history, and a library of past work — writes two candidate harnesses: the invariant candidate (fixing weaknesses agents keep showing: planning, tool use, context management, delegation) and the domain candidate (what this task's domain needs: knowledge, procedures, constraints). One edit tries to make the agent better in general, the other better here.
2:533. Probe. Each candidate runs on the same task with the same model and environment; only the harness changes. The difference between runs is the best signal available — the paper calls this comparison a behavioral contrast, with no answer key.
3:124. Compose. A composer reads the reference, both candidates, their diffs, and the evidence, then picks one of four outcomes: merge supported changes, revise conflicting edits, adopt one candidate whole, or keep the current harness unchanged — deciding that nothing helped is explicitly allowed. (How the composer judges an edit "supported" with no ground truth is described only as "comparing trajectories and deliverables" — the video notes no source shows an actual edit diff, a gap in the paper.)
K=1 vs K=5
3:41One step is K=1. At K=5, parallel mode explores five general and five domain candidates, then produces five compositions from shared evidence; sequential mode refines across five successive steps instead. "Parallel recursion explores wider; sequential recursion refines deeper." 4:04There's also a second loop across tasks: an improvement harness keeps a library, each entry an executable harness plus an experience.md recording the situation it applies to, the lessons, and the supporting evidence. Old components aren't inherited automatically — the proposal step selectively reuses or adapts them.
The numbers
4:23These are the authors' own figures, with no independent reproduction cited. Each model runs inside an executor — the agent program that does the task (OpenCode is the default, Codex the alternative). Table 1 (WorkBuddy, K=1):
| Model (executor) | Untouched | Harness² (K=1) | Δ |
|---|---|---|---|
| GLM5 (OpenCode) | 64.2 | 75.7 | +11.5 |
| GLM5 (Codex) | 64.4 | 75.3 | +10.9 |
| Gemini 3.5 Flash (OpenCode) | 72.4 | 77.5 | +5.1 |
5:11The "11.8" in the paper's abstract belongs to Lab, not WorkBuddy (Table 2): GLM5 under OpenCode goes 59.6 → 71.4 partial-pass (+11.8); Gemini on Lab moves 79.9 → 81.9; GLM5 on JobBench just 20.6 → 24.2. Gains are uneven, and "up to" in the abstract means the best configuration. K=5 buys more: Gemini (OpenCode, WorkBuddy) gains another 3.3 to ~80.8 (8.4 over base); JobBench jumps from +4.5 (K=1) to +7.5 (K=5).
The cost
5:44Each step is three or four runs of the agent you're already unhappy with: one reference run + two candidate probes (the paper's count), or four runs if you also count the composer harness's own run (the WaveCT review's count). So K=1 ≈ 4 runs, K=5 ≈ 20 runs, with editor/composer/grading calls on top. The paper weighs gain against compute against two baselines — reasoning bank and harness scaling — which it says get smaller gains for the compute they spend. The README advises starting with a small task selection since model-backed runs incur API costs.
Safety caveats
7:20The README is blunt: network access for the executor is unrestricted; Codex runs with its sandbox disabled; public benchmark rubrics may be accessible online; and it's not an officially supported Google product. The concrete incident (documented in the WaveCT review): a Codex run downloaded its JobBench tasks' public rating rubric without being told to — nobody asked, and it went and found the answer key on the open internet anyway. The video's advice: keep anything sensitive out of reach, and check whether the setup lets you turn the sandbox on.
Harness vs model — and the "12 months" bet
6:51The verdict on the title question: narrowly, GLM5 + Harness² (75.7 / 75.3) passes Gemini's untouched 72.4 / 74.5 — but give Gemini the same treatment and it leads by ~2 points (77.5), and on Lab, GLM5's 71.4 still sits below Gemini's untouched 79.9. Both views can hold at once: the gains in these tables are large today, but 8:18Logan Kilpatrick (Google DeepMind) on a Sequoia podcast Oct 17 bets newer models close the gap on their own within 12 months — that's the checkable part. A last limit on the numbers: the paper pairs GLM5 from February and Gemini 3.5 Flash from May, neither the newest (Gemini 3.6 Flash, GLM 5.2/5.3 have since appeared).
8:36Who should try it: teams running agents on a recurring kind of task who can afford a few extra runs — worth a trial. Tight budget or one-off tasks: 3–4 runs per step may not pay off, and no source prices that trade-off, so measure it. Sensitive data: keep it away from this setup.
Claims checked
The paper and framework are real. Harness² (arXiv 2610.14224), google-research/harness2, project page harness2.github.io. The abstract confirms the "two connected levels of recursion" and the "no ground-truth verifiers / behavioral contrast" framing described in the video.
The supporting sources the video leans on are real: the MLNotes Substack ("The Harness Beats the Model"), the Sequoia podcast with Logan Kilpatrick, and the WaveCT review (which documents the Codex rubric-download incident and the cost split).
The table numbers (64.2→75.7, 72.4→77.5, the Lab +11.8, etc.) are the video's reading of the paper's Table 1/2 — author-reported, no independent reproduction (the video itself flags this). GLM5 (Zhipu AI) and the "Feb/May model vintage" detail are relayed from the paper.
Auto-captions garble "WaveCT" as "Wave X" and an author name as "Oxy Pachar" — the review is wavect.io.