Why Software Factories Fail: Harness Engineering Is Not Enough

The Narrative: "You're the Bottleneck, Just Ship More"

The prevailing story in agentic coding, as Dex Horthy frames it, is seductive: StrongDM built a "lights-out" software factory where nobody even reads the code, and the lesson we're told to draw is that you are the bottleneck — the models are good enough, code is free, so just spend more tokens and ship more stuff.

Dex's whole talk is a challenge to that narrative. He's here to convince the room that the lights-out factory "does not work," and that the reason has nothing to do with skill, scale, or harness quality — and everything to do with how coding models are trained.

The Cracks Are Showing

Even as the "ship more" message spreads, the cracks are visible. Mario at AI Engineer Europe begged teams to slow down, because companies "that should not be having outages because of coding agents are having outages due to coding agent mishaps." Codebases are falling apart faster than ever.

He cites Faros AI's report from earlier in the year: since everyone picked up AI coding tools, pull-request review quality is way down — more comments, longer comments, and a rising number of PRs merged with no review at all. Incidents are up, and bugs per developer are up. The standard retort is "you're holding it wrong." Dex's reply: maybe you are — but that's not the point.

The Thesis: The Harness Is Not Enough

The core claim, stated plainly: this is not a skill issue. No amount of harness engineering, loop-maxing, adversarial PR-bot prompts, or extra tokens can solve what is fundamentally a model training issue. The magic words and review agents might raise the floor, but they can't fix the root cause.

To understand why, Dex takes the audience through how coding models are actually trained, what the current benchmarks miss, and what better ones might look like — then lands on the practical path: "we're stuck reading the code, but we can still move pretty fast."

A Brief History of the Software Factory

A useful aside: the term "software factory" was defined at a NATO conference in 1968 — the same gathering that coined "software engineering." In the 2022 pre-AI factory, humans did everything: engineers and PMs put work in a tracker, someone grabbed a task and built it, tests ran, a PR was opened, and — crucially — a human reviewed and tested the change before it shipped. When something broke, a human got paged at 3 a.m.

The insight Dex draws out is that the "build" step usually takes hours or days, and the "review" step takes hours or days too. Decades ago teams learned to add upfront planning — architecture proposals, sprint planning — precisely to reduce the chance of rework and cut the time spent reviewing every line.

The Lights-Off Factory — and Why It Failed

The agentic factory replaces "someone builds the thing" with "an agent builds the thing" — orchestration, harness, sandbox, model, computer use. The build step collapses to minutes or hours, but the human review/test step still takes hours or days, so it becomes the bottleneck. The tempting escalation is to route everything — incidents, user feedback — straight into the factory, and eventually to turn the lights off: stop reading the code entirely, since it's "going great."

Dex's aside is important for scope: this has nothing to do with vibe coding. He quotes Addy Osmani verbatim — a developer vibe-coding a side project and a team keeping a 10-year-old enterprise system alive "share almost no constraints worth naming," and most internet advice is one group telling the other how to live. HumanLayer cares specifically about brownfield codebases — and Dex warns that agents start to struggle after just 3–6 months at modern ship pace.

The conclusion: models have a real shortcoming — they can't maintain and improve codebase quality over time, not without significant human steering.

Models Can't Maintain Codebase Quality

By "maintainability" Dex means Martin Fowler's classic code smell: shotgun surgery — the point where making a change in one part of the codebase requires breaking changes elsewhere. (He gestures at John Ousterhout's A Philosophy of Software Design as the deeper reference.)

Why can't models do this? They've gotten dramatically better at one-off problems since 2024–2025, but not at improving codebase quality — a claim he admits he can't yet prove, because "there are no good benchmarks for a model's ability to maintain codebase quality." The anecdotal consensus from engineers who've used coding agents for a while: they "generally make things worse over time and make the codebase harder to work in."

Why Claude Code Won

The explanation Dex offers for Claude Code's rise is the crux of the whole talk. Great CLI agents existed before it — Aider, CodeBuff — with the exact same tools: read, write, edit, grep, bash. The difference wasn't the harness. It was that this was the first time a model lab trained a model against the harness it ships in. The model got really good at calling those tools in an agentic loop because it was reinforced to.

He relays the OpenAI team's November talk: if you're a harness builder who doesn't own the model weights and can't RL the model in your harness, you'll always be at a disadvantage against someone who owns both. Then he walks through what coding-agent RL actually looks like: generate traces, score them on "did the test pass, without breaking anything else" (binary 1/0 rewards like SweetBench multilingual), and reinforce.

The deeper problem: verifying maintainability is orders of magnitude harder than verifying "code runs and tests pass," because the cost of bad architecture is measured in months and years. If a model vibes too hard on an episode, you only find out months later — and it's nearly impossible to propagate that reward signal back across the gap.

Verifying Maintainability: Better Benchmarks

The frontier is moving, slowly. Dex walks through the emerging benchmarks that try to measure maintainability rather than just test-passing:

• Sweep Marathon (Abundant AI) — ~400-hour tasks like "clone all of Microsoft Excel," with sophisticated reward channels.
• Deep Sweep (Data Curve) — large tasks on OSS repos that were never actually built, so they can't be in the training set.
• Frontier Code (Cognition) — multi-PR tasks that penalize models whose tests don't fail on pre-patch code, plus a judge model checking code-quality rules.

His caution on judge models: they can only go so far, because "if the model knew what good code looks like, it would probably write it in the first place." Review agents and more tokens raise the floor, but the ceiling is still set by what you can teach during RL.

Turning the Lights Back On: Plan Up Front

So the pragmatic fix is to turn the lights back on and plan up front — and use AI to make the planning cheap. The sequence he recommends:

• Product review — understand the problem, desired behavior, maybe mock-ups. (Small stuff still goes straight to the agent.)
• System architecture — component contracts, data models, constraints, and how systems fit together.
• Program design — the step he thinks is most under-emphasized: types, method signatures, program layout, call stacks. He credits Dylan Mulroy at Cloudflare for using call graphs as part of planning.
• Vertical slices — the order of implementation, multi-repo coordination, and how to check progress along the way.

Closing: Too Many Bad PRs

His counterintuitive diagnosis of PR overwhelm: you don't have too many PRs — you have too many bad PRs. A good PR is a joy to read; but a PR that needs even 20% rework is "an emotional and intellectual burden on both the reviewer and the submitter." With model-assisted planning and alignment, the alignment is shorter, review is faster, and coding is faster — so you're genuinely moving faster while still reading everything and owning the code.

The closing note is honest and encouraging: it's easy to be bummed that we can't just YOLO everything and never read code again — but "we're engineers, and these are just constraints." Models are good at some things and not others; figure out how to solve problems within the constraints. Use loops, seek leverage, and go solve hard problems.

Sources & Claims Checked

• 🔗 humanlayer/12-factor-agents — Dex Horthy's repo on principles for production-grade LLM software (verified live: ~26.5k stars, TypeScript).
• 🔗 Official AIE talk page — corrected transcript, summary, and resources for this talk.

Claims checked:

• ✅ "Software factory" coined at a NATO conference in 1968 — historically accurate; the 1968 NATO Software Engineering Conference (Garmisch) coined both "software engineering" and discussed the "software factory."
• ✅ Speaker identity — Dex Horthy, co-founder of HumanLayer (humanlayer.com), author of 12-factor-agents; the repo checks out.
• ⚠️ Claude Code revenue figures. The speaker cites Claude Code going "from nothing to $4 billion, now $9 billion in under a year." Verified public figures are more nuanced: Claude Code specifically hit ~$1B annualized within six months and ~$2.5B run-rate by Feb 2026, while Anthropic's overall run-rate crossed ~$9B at end of 2025 and ~$65B by July 2026. The "nothing to billions" story is real; the precise split is loose and appears to conflate Claude Code with Anthropic total.
• ✅ Martin Fowler's "shotgun surgery" — a real code smell from Refactoring.
• ℹ️ John Ousterhout (transcript renders "Osterhout") — author of A Philosophy of Software Design, referenced correctly.

Benchmarks named (SweetBench multilingual, Sweep Marathon, Deep Sweep, Frontier Code) are presented as the speaker's examples; SweetBench and Frontier Code are established, while Sweep Marathon and Deep Sweep are newer and cited here as he described them.