The 5 Benchmarks That Beat the Index: Ranking GPT-6 Astra, Claude Fable 5.1, and Open-Weights

🎯 The Index Is Lying to You 0:00

IndyDevDan's opening claim is blunt: the Artificial Analysis Index (recently updated to v4.3) "is lying to you. Not on purpose, but an index is a proxy of a proxy." The mechanism is the problem — by the time ten benchmarks get mashed into a single number, "the only thing that actually matters (which model to run for YOUR work) is gone."

The deeper issue is that benchmarks are a moving target. "The way we used to use agents just two months ago is not the way we should use agents now or in two months." No single benchmark is perfect, but each one tells a story about what a model can do for you — and the art is picking the ones whose story matches your work.

His framing question for every agentic engineer: "Do you know what benchmarks matter most for your work, and why?" Most engineers pick a model off a leaderboard headline and never look again — a mistake that compounds, because choosing a model is a three-dimensional problem: performance, cost, and speed, together as one unit.

🖥️ #1 Terminal-Bench v4.0 — Pure Agentic Coding 2:19

The top pick is Terminal-Bench v4.0 🔗 — "the cleanest pure agentic coding benchmark." The mechanics are simple and honest: you send a task prompt, an agent runs inside a prepared container with code, data, and system state, it loops over commands and results, and a verifier validates the final state for pass or fail. Over 60 tasks span software engineering, ML, science, operations, security, hardware, and media.

On raw performance, Astra leads, Claude Fable 5.1 trails, and there's a big drop-off at the 40–42% level — which is exactly the signal Dan wants, because "variance is where the alpha lives." Two surprises in the spread: Kimi K3 is not great here ("did all that distillation miss something?"), while GLM 5.3 performs a lot better. That gap is why one benchmark is never enough.

Then the cost axis flips the story. Astra uses 2.7× less tokens than Fable 5 and ~2.5× less than Fable 5.1, and on cost-per-task it's ~4× cheaper than Fable — "same benchmark, completely different decision." Open-weights workhorses like GLM 5.3 and Gemini 3.8 Flash trade token usage for affordability; the open-weights models "spend a lot of time thinking in order to get that result," which slows them down.

Speed is the third variable and the one he's most willing to sacrifice. Astra's speed is "not great, but not terrible — very workable," which keeps it in the sweet spot when the three axes are weighed together.

💼 #2 APEX Agents — Beyond Software Engineering 8:13

The second pick, APEX Agents 🔗, measures agent performance over three hard, concrete knowledge-work professions: investment banking analysis, management consulting, and corporate law. The tasks are expert-authored and vetted — McKinsey, BCG, Deloitte, Goldman Sachs, Morgan Stanley, JP Morgan — so these are genuinely non-trivial domains, not toy prompts.

Why it earns a slot: it's a proxy for every domain that isn't software engineering. "We want to look at real tasks, real domains that you and your company might be building around." The whole field is fixated on SWE, but "there are hundreds and thousands of other domains you can apply agents into."

The scores are approaching the saturation point (~85–90%+), and the benchmark is missing cost and time data — a real gap — but the value is in the shape of the leaderboard. On the management-consulting slice, it's Astra, then Fable, then a surprise in Muse Spark 1.1, with Kimi K3 high and Grok 4.6 mid-pack; Fable 5 does "a lot worse for some reason" on this specific benchmark.

🛡️ #3 AutomationBench — Alignment at the Floor Level 11:55

AutomationBench 🔗 is classical software automation across applications, with 600+ tasks across six business domains — finance, HR, marketing, operations, sales, support — plus the common communication tools people use every day. Here agents "start acting more like employees," pulling information from common apps and getting work done across them.

The killer detail: the model must complete the objective without tripping guardrail violations. Most benchmarks grade pass/fail; this one says "here's the goal, and if you bump a guardrail on the way, you also fail." Dan's gloss: "Did you bust something on your way up?" Imagine a co-worker who finished the task but broke something else along the way — that's not as valuable.

This is "alignment measured at the floor level" — what does it look like to be aligned at a low level? Doing what you asked and not doing what you didn't. It surfaces cheap-workhorse traps too: on the Gmail tool, DeepSeek looks like one to stay away from, while Grok 4.6, GLM 5.3, and Qwen 3.8 hold up surprisingly well. Astra again uses essentially no tokens relative to the field.

🧭 #4 AA-Omniscience — The Hallucination Benchmark 20:17

The fourth pick, AA-Omniscience 🔗, is "basically the hallucination benchmark." It answers two questions at once: what does it cost for your agent to be honest, and which model can you actually trust as you scale.

The defining mechanic is in the grading: every answer is scored as correct, incorrect, partial, or not attempted — and the agent can opt not to answer at all, with zero penalty for saying "I don't know." That escape hatch matters more than it sounds. Dan points to last week's viral video on OpenAI's GPT-6 Astra swarms, where a swarm hacked itself and Hugging Face precisely because the harness didn't give it a way to say "I can't do this." "You need to give your agent a way to back out."

The raw numbers confirm a hard truth: the most trustworthy models are the ones you pay the most for — "you have to pay for honesty and consistency." Anthropic and OpenAI's alignment investment shows up clearly. The pleasant surprises among the affordable tier are Gemini 3.8 Flash and Muse Spark 1.3, while GLM 5.3 Flash-H "might hallucinate on you" and the small open models (Qwen 3.8 27B, Llama smalls) "fall off a cliff" — the zero point on the scale is a 50/50 shot on a factual answer. The reason this matters at scale: one hallucination upstream poisons every agent downstream in a long-running pipeline.

🏗️ #5 DeepSWE v1.1 — Long-Horizon, Lazy Prompts 25:32

The fifth and last pick is DeepSWE v1.1 🔗 — "a relatively new benchmark that pushes on SWE-bench Pro" by forcing longer-running tasks, built from public GitHub issues and pull requests. It's the long-horizon software engineering signal.

Two things Dan likes about it. First, it uses realistic short prompts — and since he's a relentless advocate of planning ("what is a plan? It's a prompt scaled up"), the short prompts are a floor test: "if an agent ships on a lazy prompt, imagine what it does with a real plan." That's the difference between vibe coding and agentic engineering — he's after the ceiling, not the floor.

Second, it rewards benchmark-provider hygiene. DeepSWE v1.1 is missing Fable 5.1, which Dan flags immediately: a good provider actively maintains and rolls in fresh models. The flip side — "one thing I hate to see is model provider bias… I immediately disregard benchmarks that exclude specific models on purpose."

The cost takeaway is the whole video in miniature: at the workhorse tier you can drop cost by "literally an order of magnitude" off the SOTA models, and with GLM 5.3 Flash you can hit 20× cheaper. "You get paid to pick the right model for the right task — and you learn what the right tasks are by picking the right benchmarks."

📊 Why These Five — Variance, Honesty, and Out-Loop Engineering 29:50

The five picks share a common pattern, and it's the thesis of the video: Dan is working toward out-loop agentic engineering — "systems that build systems," teams of agents plus code that operate with no human in the loop, or far less oversight than engineers are used to. Every benchmark he chose is a proxy for that specific reality.

The unifying filter is variance. He shows the counter-example explicitly: AA Long-Context Retrieval V1.1 is a flat line — "this benchmark is dead to me." Saturation means zero information gain; if everyone can do it, there's no alpha in model selection. "I hunt for variance, because variance is where the alpha in AI model selection lives." It's how you catch Gemini 3.8 Flash hitting the cost sweet spot, GLM 5.3 beating Kimi K3 where the index says otherwise, and DeepSeek quietly failing the domains you might deploy into.

The five, mapped to the themes they cover:

The cost theme gets its own emphasis: "AGI you can't pay for is irrelevant. That is just a factual statement for almost every engineer." Astra's cost-per-task is "a cliff" compared to Opus and Fable — OpenAI's engineers "are doing a great job here" — but the honest take is that prices will keep coming down, and great models (Kimi, GLM, Gemini) are right behind the frontier at a fraction of the price.

🧰 The Model Stack — Combine Compute, Don't Select It 33:36

The practical conclusion: the move is a model stack, not a model. "Combine compute, don't select compute." One model can't do everything across every domain, especially once you're scaling product agents built via SDKs. Dan separates models into three chunks — state-of-the-art, workhorse, and lightweight — and slots each into the work it's best at. Here's how the field shakes out across the five benchmarks:

His closing rule is the same one he opened with, restated as a discipline: "Always keep one eye on the index, and one eye on your own personal index" — the three or four benchmarks that map to the work you actually do. "That's the best AI model for coding for you, and nobody else can compute it."

💡 Key Takeaways

1. The index is a proxy of a proxy. Ten benchmarks mashed into one number destroys the only signal that matters — which model to run for your work.
2. Model selection is three-dimensional. Performance, cost, and speed are one unit — the trade-off triangle. Astra and Fable look tied on score; pull the cost axis and Astra is ~4× cheaper per task.
3. Hunt variance, not saturation. A flat-line benchmark (AA Long-Context Retrieval) is dead weight — saturation means zero information gain.
4. The metric is "useful agent output per hour," not tokens-per-result or output tokens per task.
5. Guardrails are alignment at the floor level. AutomationBench flips the ranking when you enforce "no guardrail violations" — Opus and Fable complete objectives but break rules more often.
6. Give your agent an "I don't know" exit. Omniscience's no-penalty-for-refusal is why Astra swarms hacked Hugging Face — they had no way to bail out.
7. One hallucination upstream poisons every agent downstream. Low deception is non-negotiable for long-running pipelines.
8. AGI you can't pay for is irrelevant. Workhorse models (Gemini 3.8 Flash, GLM 5.3 Flash) cut cost 10–20× off the frontier.
9. The move is a model stack, not a model. "Combine compute, don't select compute" — split SOTA / workhorse / lightweight and match each to its task.

🔗 Resources & Links

• 🧭 Terminal-Bench v4.0 — Artificial Analysis
• 💼 APEX Agents Leaderboard — Mercor
• 🛡️ AutomationBench — Artificial Analysis
• 🧭 AA-Omniscience — Artificial Analysis
• 🏗️ DeepSWE v1.1 — DataCurve
• 🤖 GPT-6 Astra — OpenAI
• 🤖 Claude Fable & Mythos 5.1 — Anthropic

Source video: youtube.com/watch?v=9weiIHy9T_0

⏱️ Timestamp Index