Let Your AI Agent Do Its Own Harness Engineering

0:00The strategies that make coding agents more reliable over time also translate to production agents — and the big one is harness engineering: tracking every mistake your agent makes in production, then going back to figure out what to change in its rules and skills so those mistakes don't repeat. This is a LangChain-sponsored walkthrough, but the ideas are framework-agnostic.

Harness engineering, generalized

0:33Cole's specific definition: not just "make the agent better over time," but retroactively mining production mistakes — the "biggest gold mine of information that most people ignore" — and turning them into better rules. It translates especially well because production agents are now built exactly like coding agents: the harness is managed with markdown, which keeps it flexible.

The markdown harness

1:26The one-to-one mapping from coding-agent components to production agents:

• Rules = the system prompt (instructions.md).
• Sub-agents = multi-agent workflows and delegation.
• MCP tools = connections to external systems.
• Skills = workflows, replacing a lot of hand-built tools.
• Hooks = guardrails, monitoring/blocking per tool call or at the start/end of a conversation.

4:01Building it now is a 30-line agent.py plus markdown: an instructions.md at the agent root is picked up automatically (like CLAUDE.md), skills live in a skills/ folder, and the platform stitches it together. To change the agent's behavior you edit markdown, not plumbing.

Loop 1: memory

5:30The first learning loop is about the agent learning lessons while it runs — and that means escalating uncertainty to a human. When the agent struggles, it should recognize that and flag it rather than bluff. A team inbox collects what it's unsure about; a human gives the right answer; the agent writes it down as a lesson. Those lessons accumulate in a context hub (a searchable knowledge base) plus a distilled agents.md of core rules. LLMs hallucinate, but they're surprisingly good at surfacing their own uncertainty — and as the lessons build, less gets escalated over time.

The help-desk demo

7:52A live demo with a member help desk for Dynamous, using real questions about his Archon framework. "Nina" asks why her Archon run used old command text after committing a fix. The agent answers but surfaces uncertainty ("I don't have enough yet to say for sure") and escalates — the inbox ticks from 4 to 5. Cole confirms the fix ("Archon creates worktrees from origin main → git push and rerun"), and 10 seconds later the agent has written a new shared memory and updated Nina's private memory. 10:25Then "Omar" asks a similar but not identical question (a PR missing three commits) — and this time the agent answers confidently, extrapolating from the lesson rather than just pattern-matching the exact same question.

Loop 2: retroactive fixes

11:40The second loop targets the higher-level harness — the core workflows and instructions — not the per-case lessons. Tracing and metrics surface real production failures with evidence (the exact trace), and an engine proposes a concrete fix to instructions.md (surveying rules, guardrails, skills). It's the same "system evolution" idea applied to production agents. Cole clicks apply to approve the change — with version commits, promote, and rollback, so a bad fix isn't the end of the world.

4/14 → 13/14

14:54On a small benchmark of 14 real questions, the initial agent got 4 of 14 right; after going through both learning loops (lessons + engine-suggested improvements), it reached 13 of 14. The caveat he states plainly: LLMs won't be 100%, but you can get very close — and the two loops are what get you there.

Human in the loop

15:23The through-line: you want the agent to surface and propose fixes automatically, but you stay in the loop approving everything. Cole explicitly warns against letting an agent autonomously improve itself — "I lose control of my agent, and I don't really know the direction it's heading." Harness engineering is not a one-time job; it's a continuous loop with you at the controls.

Claims checked

Sources & further reading