The Best AI Agents Need Less Code Than You Think

1. What Is LangSmith Engine?

⏱ 0:00

LangSmith Engine is an agent for agent engineers. It's an always-on autonomous system that automates the core improvement loop: find problems in your agent's traces → cluster similar issues together → generate code fixes → suggest evaluations. Think of it as a tireless teammate that watches your agent 24/7 and opens PRs when things go wrong.

2. LangSmith 101

⏱ 1:25

LangSmith is the observability platform for agents. It provides the foundational tooling Engine builds on:

Traces — full execution logs of every agent run
Evaluations — automated scoring of agent outputs
Annotation queues — human review workflows
Playground — interactive prompt testing environment

3. Engine Under the Hood: A Deep Agent

⏱ 3:49

Engine is built on the LangChain Deep Agents harness. It ingests production traces, clusters issues by similarity, proposes pull requests with fixes, and suggests new evaluation examples. The system is genuinely autonomous — it operates continuously without human prompting, making it a true "deep agent" rather than a simple chain or workflow.

4. Clustering Millions of Traces

⏱ 6:08

To handle massive trace volumes cost-effectively, Engine uses three-level trace views purpose-built for the system:

Ultra-condensed stats — high-level metrics and aggregations
Messages view — intermediate detail showing conversation flow
Full trace — complete execution details when deep investigation is needed

This tiered approach made LangSmith truly "agent-native" — designed for machine consumption, not just human dashboards.

5. Handing More Control to the Agent

⏱ 10:10

A key lesson from building Engine: strip back deterministic scripts and give agents tools + freedom. Early versions used rigid, pre-filtering deterministic pipelines. Over time, the team removed scaffolding and handed the agent CLI tools with prompts guiding behavior instead of hard-coded scripts. The principle: guide with prompts, not scripts.

6. Sandbox as a Tool

⏱ 13:21

An important architectural insight: the sandbox is a tool, not an environment. Engine runs inside LangSmith Deployment and calls OUT to LangSmith Sandbox when it needs to execute code. The sandbox is invoked on-demand as just another tool in the agent's toolkit, not the container the agent lives in.

7. Four Sub-Agents & The Org Chart

⏱ 14:11

Engine's architecture mirrors a corporate org chart with specialized sub-agents:

Main Agent (Opus) — the manager, orchestrating the full workflow
Screener (Haiku) — fast, cheap model that reads full traces to filter signal from noise
Verifier — quick-check agent confirming issues are real
Issue Creator — writes up confirmed issues with context
Fix Generator — proposes code changes to resolve issues

8. Evals: IssueBench, Harbor & Shadow Production

⏱ 16:51

Engine's evaluation strategy is multi-layered:

IssueBench — ~50 curated tasks for offline benchmarking
Harbor framework — synthetic environments for controlled testing
Stub server — mocks LangSmith API endpoints for isolated testing
Shadow production — forks real production traces but doesn't create actual issues; tests Engine's behavior against live data without side effects

9. From Noisy PRs to Issue Inbox

⏱ 23:05

V1 of Engine spammed noisy pull requests — too many, too low quality. The team evolved the UX into an issue inbox: clustered issues grouped by pattern, with history and frequency tracking. This introduced human-in-the-loop review with natural language feedback, letting engineers guide Engine's priorities without touching code.

10. Engine's Memory: Agent Overview Document

⏱ 25:56

Engine maintains an Agent Overview Document — similar to a CLAUDE.md file. This living document is referenced at the start of every run and updated as Engine learns. It stores user preferences, codebase structure, priority areas, and past decisions. This creates a hybrid memory system: persistent knowledge that evolves with each interaction.

11. Cost Control & Model Selection

⏱ 29:25

Engine uses a multi-model cocktail optimized for cost and capability:

Opus — main orchestrator (highest capability)
Haiku — screener agents (fast, cheap, high-volume)
GPT 5.5, Gemini, open-source — mixed in for specific tasks

Model selection is driven by hill-climbing against evals — swap in a cheaper model, check if eval scores hold, keep if they do.

12. Engine on Engine: Self-Improvement Loop

⏱ 48:49

The ultimate recursion: Engine runs on its own traces. The team initially assumed this would be terrible, but it's becoming the primary improvement method. Engine identifies its own failures, clusters them, and suggests fixes to itself. The missing piece: branched agent testing — the ability to test proposed fixes on a branch before deploying to production.

🎯 Key Takeaways

LangSmith Engine is an always-on agent that finds failures, clusters issues, and opens fix PRs
Architecture mirrors an org chart: Opus main agent → Haiku sub-agents
4 specialized sub-agents: screener, verifier, issue creator, fix generator
Three-level trace views enable cost-efficient analysis at scale
Strip back deterministic workflows — give agents tools + freedom, guide with prompts
Sandbox is a tool the agent calls, not the environment it runs in
UX evolved from noisy PRs → issue inbox with human-in-the-loop review
Agent overview document serves as persistent memory (like CLAUDE.md)
Multi-model cost optimization via eval-driven hill-climbing
Self-improving: Engine analyzes and fixes its own traces
Hard unsolved problem: testing agent fixes before deploying them
IssueBench + Harbor for offline evals, shadow production for online testing

📚 Resources

The Best AI Agents Need Less Code Than You Think — LangSmith Engine Deep Dive