From Vibes to Production: Evaluating and Shipping AI Agents That Work

0:12Laurie Voss — co-founder of npm, now Head of Developer Relations at Arize AI — runs a packed two-hour workshop on the single most under-served part of building agents: knowing whether they actually work. The whole talk is one argument, made twice — first in theory, then hands-on. "AI features usually get shipped by building an AI feature, running a few queries, saying 'does this look right?' Yeah, it looks good. And then you ship it." That's the vibes problem, and everything after is how to replace vibes with numbers.

The vibes problem

8:54The most popular method of shipping AI today is running a few test queries, eyeballing the output, and shipping. It fails on inputs you didn't test. The usual fix for flaky software — unit tests — doesn't work here because the output is non-deterministic: the same prompt produces different text every run, and two different responses can both be correct. So teams fall back on human review, which doesn't scale, doesn't catch regressions, and doesn't run in CI. Voss lists what you literally can't do without evals: change a system prompt to fix tone without unknowingly breaking faithfulness; manually test every input combination; or switch models — "without evals, switching models means weeks of manual testing… with evals, you run the suite and know within hours." He calls it the difference between flying by the seat of your pants and flying by instruments, and names Descript, Bolt, and Anthropic's own Claude Code as teams that shipped on vibes and then had to build evals.

Evals = tests, traces = logs

7:42The core mental model: evals are testing for AI, and traces are logs for AI. Tests assert "given this input I expect this output"; evals do the same for AI outputs that are never identical twice. Logs record what your server did; traces record what your AI did — every agent call, tool call, and LLM invocation, with inputs and outputs at each step. The building blocks of a trace are spans, arranged in a tree: an LLM call is a span, a tool call is a span, and a full agent turn is a span containing other spans, each recording input, output, timing, token counts, and metadata. "Traces tell you what happened. Evals tell you whether it was any good."

Three kinds of evals

11:27Code evals are deterministic functions — "did the output parse as JSON? is it under 500 tokens? did it mention the ticker?" They run in milliseconds, cost nothing, and are fully reproducible, but like unit tests they can be brittle. 12:14LLM-as-a-judge evals use a second LLM to grade outputs against a rubric (rules about what counts as good). They're flexible enough for semantic questions — factual accuracy, faithfulness to sources, tone — because they understand meaning, not just strings. Their weakness: they cost money and are themselves non-deterministic, so they must be calibrated against human judgment. The third type is human evaluation, the gold standard that's slow and expensive — and, Voss notes, "human annotators miss up to 50% of defects due to fatigue," so humans aren't perfect judges either. Most real systems use all three.

Why agents are harder

14:27Agents add a ladder of complexity. A single LLM call has one input and one output; an agent makes a series of tool calls and decisions, so at every step the chance of a mistake compounds. You have to judge intermediate steps too — did it use the right tool, pass the right arguments, know when to stop (loops are a real failure mode)? Multi-agent systems add handoffs on top.

15:38His cascading-failure example is a classic: ask about "Tesla" (the car company), and the agent searches the web, finds information about an 18th-century inventor, and writes a beautiful, well-formatted report about that inventor. "Nothing that it did there was wrong" — it searched for Tesla, got Tesla, and wrote about Tesla. There's no unit test that would catch it. 16:19The reverse also happens: on τ²-bench (Anthropic's multi-turn tool-use benchmark), an agent was told to reschedule an economy-class flight — thought impossible — but the policies it was given let it upgrade the flight to first class, which can be rescheduled. It solved the "impossible" task and failed the test. "Creatively correct" is a category you have to account for.

17:25He also splits evals into capability evals ("can my agent do this at all?" — expected to mostly fail, a hill to climb) and regression evals ("does it still do what it used to?" — expected to pass every time). Mature agents accumulate capability evals that slowly convert into a regression suite.

Read traces before writing evals

38:54The most important practice in the entire workshop, in one instruction: "Read your traces before you write evals." Read a dozen or more end-to-end, focusing on the ones that went wrong, and figure out what specifically broke before writing a single evaluator. If you automate before you understand your failures, you'll measure what's easy to measure instead of what matters. You also need requirements first — you can't say "it doesn't work" until you've defined what "works" means. And the definition of good doesn't live with engineers: it lives with PMs, QA, and support. "OpenAI put it really nicely: people management skills are AI skills."

Coding your failures

45:37The structured way to read traces comes from qualitative research: open coding (read each trace, write down what you see in natural language — don't invent categories in advance) then axial coding (group those into categories). Each root cause points to a different fix: bad search results is a retrieval failure, right data + wrong conclusion is a reasoning error, made-up prices are hallucination, out-of-domain answers are scope violations. Then prioritize by severity × frequency — a one-in-a-hundred "how to make a bomb" answer gets fixed before a common-but-mild formatting issue. 48:36And the Swiss cheese model: each eval layer has holes, but stacked they don't line up — code evals catch format issues, LLM judges catch reasoning gaps, humans catch subtle hallucinations. "Stack the cheese."

Instrumenting in two lines

25:36The demo uses a financial-analysis agent built on Anthropic's Claude Agent SDK, instrumented with Arize AX (the speaker's employer — flagged as such below). The key architectural point is vendor-agnostic: instrumentation rides on OpenTelemetry (the industry-standard observability layer) plus OpenInference (which adds LLM-specific attributes — prompt text, token counts, model, tools). Because the SDK authors already wrote the OpenInference hooks, you only add two lines — point it at a collector, name the project — and every LLM call and tool call emits a span. "You can't evaluate what you can't observe, so observation comes first."

Code evals

49:23The first eval is almost embarrassingly simple: does the requested stock ticker actually appear in the report? Five lines of Python that look for the uppercase acronym. The ticker check caught a real, subtle failure — a query about "Amazon AWS profitability" produced a report entirely about the AWS division that never mentioned Amazon's own ticker. 57:29The key principle: grade what the agent produced, not the path it took. Checking that it called tool A before tool B is too rigid — agents regularly find valid approaches the designer didn't anticipate, and grading the path punishes creativity. A code eval can be as complex as querying a live API to verify a stock price; the only requirement is deterministic grading.

Built-in LLM evals

58:17Every LLM-as-judge has three parts: a judge model (often bigger/slower than the worker — Sonnet judging Haiku's output), a prompt template (the rubric), and the data. Arize AX ships built-ins like correctness, faithfulness, tool selection, and refusal detection. Then the demo delivers its best lesson: 1:01:07the correctness eval scored 0 out of 13 — not because the agent was wrong, but because the judge's training data stops in January while the agent was reporting live 2026 stock data. Correctness relies on the judge's own knowledge; faithfulness gives the judge the same source context the agent had, putting them on a level playing field. 1:04:21Faithfulness reported 7 unfaithful, 6 faithful — roughly half the reports hallucinated things not in the sources. That's a useful capability eval: a hill to climb. "The difference isn't that one eval is better — the difference is the use case."

Writing a custom rubric

1:06:25Built-ins can't check "is this report actionable?" So you write a custom rubric, with four parts: define the judge's role (give it context about what it's looking at — the "expert financial analyst" trope barely matters), list explicit pass/fail criteria (the most under-invested part — "helpful and accurate" is an aspiration, not a criterion; "contains specific recommendations" is observable), label data with XML tags (so the judge doesn't confuse instructions with input/output), and define output choices outside the prompt (binary pass/fail beats a 1–5 scale; don't tell the judge to "spit out correct or incorrect"). Voss controversially adds examples of good and bad output — LLMs are great at copying an example, though some practitioners warn it overfits.

1:17:40Three anti-patterns: treat eval prompts like code (version them, test them — if the judge disagrees with your labels 40% of the time, your eval is wrong); the god evaluator (one mega-eval checking six things is a calibration nightmare — "six things is six evaluators, not one"); and know your guardrails vs northstar metrics (a hallucinated stock price is a hard ship-blocker; "always recommend complimentary investments" is an aspirational nice-to-have).

Can you trust your judges?

1:20:06The judge is just a classifier, so you evaluate it like one — against human ground truth. 1:21:52You build a golden data set by having humans (often non-technical domain experts, in the UI) label outputs, then split it 75/25 into dev/test so the judge doesn't overfit. The most valuable output of meta-evaluation is the disagreements — they usually reveal an ambiguity in your rubric that you tighten ("includes forward-looking analysis" → "…with specific recommendations or guidance").

1:27:46Known judge biases: position bias (favors whichever option comes first), length bias (scores longer responses higher even if it's filler), confidence bias (fooled by confident tone, just like humans), and self-preference bias (likes its own outputs — use a different model or lab for the judge than the worker). 1:29:06The benchmark is human performance: inter-rater reliability between two humans is often as low as 0.2–0.3, so a judge that disagrees with you some of the time may be fine — one that disagrees all the time is the danger signal. And "failures should seem fair": if a failing trace looks fine to you, the eval is probably the problem, not the agent.

Closing the loop

1:30:26Now the payoff. Save failing traces as a data set, then hand the explanations to a coding agent. 1:33:11"Nothing here is guessed": Claude reads the judge's explanations and rewrites the prompts so each change traces back to a specific failure — if the judge said "lacks specific recommendations," the prompt gets rewritten to require them. That's data-driven prompt engineering, automated. 1:35:22Then you run an experiment: same data set, same evaluators, only the prompt changed, so any score difference is attributable to your change. (Voss one-shots his demo from 50% wrong to 100% right and immediately caveats that real iteration is a slow-climbing graph, not a one-shot.)

1:39:48On sample size: 12–20 examples is directional; for shipping decisions aim for 200–400 — a human-scale number, not tens of thousands. 1:40:30And the investment hierarchy: data quality > prompting > model selection > hyperparameters. Fix stale sources before you iterate on prompts; temperature and top-p are almost never the high-leverage change.

Production & where to start

1:42:59Dev experiments aren't the end — you're done when the agent keeps performing on unseen production traffic. Online evals run your ticker-check/faithfulness/actionability evaluators on incoming traces automatically, at span or trace scope, sampled (10% or 1%) because LLM judges cost money. 1:45:59The payoff compounds: every failure your online evals catch becomes a new test case, and over time you build a data set unique to your application that nobody else has. 1:48:39The closing advice: don't do it all at once. Add the two lines of tracing and just read your traces — "15 minutes of reading real outputs will teach you more than an hour of building dashboards." Then write one code eval for the thing that matters most. "Evals are infrastructure, not an afterthought."

Claims checked

Sources & further reading