1 The Harness is Data Science
Hamel opens by referencing the widely-circulated article about harnessing engineering — a team at OpenAI that used agents autonomously for months to build substantial internal software. The key insight most people miss: the "harness" isn't just specs and unit tests. It also contains logs, metrics, and traces — the entire observability stack.
Hamel's central thesis: the harness is data science, or at least a big part of it is. The skills required to keep agents on track — analyzing logs, designing metrics, validating model outputs — are classic data science competencies that the AI engineering community has been neglecting.
2 ML Engineering Then vs AI Engineering Now
Four years ago in ML engineering and data science, practitioners would:
- Examine data carefully — visualize it, understand distributions
- Align predictions with human labels — rigorous validation against ground truth
- Use the right tool for the task — deliberate model selection
- Design metrics aligned with business goals — thoughtful metric engineering
AI engineering today has taken a step back. Now teams rely on "vibes" to know if they're doing the right thing. They ask the same model (or another model) whether it did a good job. Metric design is often skipped entirely — people use an LLM to grade on a 1-to-100 scale, or grab someone's metric package off the shelf without thinking.
The data science pieces of AI engineering — evals and retrieval — have "died" so many times Hamel can't count. Engineers are often scared of what they don't understand, and these areas keep getting deprioritized.
3 Mistake 1: Generic Off-the-Shelf Metrics
Shreya takes over to walk through the five most common mistakes. The first: using generic or off-the-shelf metrics like "helpfulness," "hallucination," or "coherence" to measure agent accuracy.
While these sound important, they have two critical problems:
- Ambiguity — "hallucination" means different things in different contexts. A medical app and a legal app have completely different definitions of what constitutes a hallucination
- Non-transferability — different applications use different tools and data, making a shared evaluator meaningless
The Data Scientist Fix: Explore Your Data
Instead of reaching for off-the-shelf metrics, a data scientist would:
- Build custom interfaces — use tools like Codex, Claude Code, or Cursor to build annotation UIs
- Load up traces and read each message in each trace one by one
- Pretend you're the user — discuss with PMs what the actual mistakes are
- Write open notes about failures, then categorize them into bespoke failure modes
Example: in a real estate agent tour app, the bespoke failure modes were things like "hallucinating tour times" — something completely specific to that app that no generic metric would catch.
4 Mistake 2: Blindly Trusting LLM Judges
The second mistake is using LLM judges without validating the validators. Teams find a failure mode, then simply ask an LLM "how often is this occurring in my data?" without any notion of trust in the judge itself.
Common anti-pattern: asking an LLM to rate quality on a 1-to-5 scale, getting a histogram of mostly 1s and 5s, then trying to make business decisions from that. Those numbers are neither trustworthy nor actionable.
The Data Scientist Fix: Treat LLM Judges as Classifiers
A data scientist would recognize that an LLM judge is an imperfect classifier and apply standard ML validation techniques:
- Partition labeled traces into train/dev/test splits — for each failure mode
- Iterate on the prompt/model using the train and development sets
- Validate on a held-out test set to ensure you're not overfitting the judge
Critically, LLM judging is often an imbalanced classification task — only a small fraction of traces actually exhibit a given failure. Don't just measure accuracy. Use metrics designed for imbalanced problems:
- Precision and recall
- False positives and false negatives
5 Mistake 3: Bad Experimental Design — Synthetic Data
The third mistake covers bad experimental design, starting with synthetic data generation. Teams generate synthetic data by simply asking an LLM for "five generic questions" — and all the traces end up looking essentially the same. No diversity, no coverage of edge cases.
The Data Scientist Fix: Systematic Generation
Rather than letting the LLM drive the entire generation process, use LLMs for very small parts of it:
- Hypothesize which dimensions vary in the data users bring to the system
- Generate combinations of these dimensions systematically
- Review all synthetic data for quality — look at it, ensure good diversity
A practical exercise: examine real traces and identify at least three dimensions that vary across users — for example, user persona (novice vs expert), query complexity, topic domain. Use LLMs to generate values for each dimension, then take the cross product to produce synthetic data with proper coverage.
6 Metric Design: Binary Over Scales
The second facet of bad experimental design is poor metric design. Teams commonly score things on a 1-to-5 or 1-to-100 scale, which is neither interpretable nor actionable.
The Data Scientist Fix: Binary Classification
Make the problem as least complex as possible:
- Use binary classification — does this fail or not? Pass or fail. Not a 100-point scale
- Align with business outcomes — metrics should drive decisions, not just fill dashboards
- Make LLM judges narrow in scope — a judge for one specific failure mode, with a binary output
- Label lots of data yourself to measure alignment
Shreya and Hamel emphasize that nobody writes a good LLM judge prompt on the first try. It's an iterative process — but binary framing dramatically reduces the search space for getting it right.
7 Mistake 4: Wrong People Labeling Data
A surprisingly common failure: teams outsource data labeling to AI engineers or developers who lack domain expertise. Unless you're building a coding app, the developers typically don't have the contextual knowledge to judge whether outputs are correct.
The Data Scientist Fix: Domain Expertise + Criteria Drift Awareness
- Don't trust labels blindly — verify the labelers have genuine domain expertise
- Look at the labels yourself — unexpected problems emerge that you can't anticipate
- Watch for criteria drift — from the paper "Who Validates the Validators" (Shreya is a co-author), people don't actually know what they want until they look at some data
The idea that you can define a rubric once and be done is a trap. Criteria evolve as humans encounter more examples — this is criteria drift, and it means you must force reviewers to continuously examine real data and refine their standards.
8 Mistake 5: Automating Too Much
The final major mistake: thinking "can't Claude just do it for me?"
The answer is no. LLMs can catch obvious errors and low-hanging fruit, but they can't read your mind. They don't know all the product nuances that can go wrong. The most important failures — the product failures that actually matter to users — require human context that you need to externalize explicitly.
9 Other Common Pitfalls
Hamel rapid-fires through additional pitfalls they've seen across organizations:
- Misusing similarity scores — ROUGE, BLEU, and similar metrics appear on eval dashboards but are rarely meaningful for LLM applications. Do you actually want to measure text similarity?
- "Is this helpful?" as a judge prompt — generic, non-specific judge prompts that aren't tailored to the product's actual quality criteria
- Making annotators read raw JSON — all friction should be removed from data review. Build custom annotation interfaces that make reading data delightful
- Reporting uncalibrated scores — LLM judge scores without studying alignment against human judgment are meaningless
- Ignoring criteria drift — failing to account for how labeling standards evolve over time
- Overfitting judges to data — just like in ML, don't hill-climb against a single dataset. Set aside held-out data to verify generalization
- Ineffective data sampling — not thinking carefully about what data to evaluate
- Dashboard metrics without signal — every metric on a dashboard should earn its place with actual actionable signal
10 The Data Science Corollary
Hamel maps each AI engineering challenge back to a classical data science skill:
- Error/data analysis → EDA (Exploratory Data Analysis) — finding patterns in traces, reading and analyzing them systematically
- Good scoped metrics → Metric design — application-specific, aligned with real problems
- LLM judge alignment → Model validation — ensuring the judge agrees with human judgment, just like validating any ML model
- Test data quality → Data curation — systematic curation of evaluation datasets
- Production monitoring → Observability — continuous monitoring of deployed systems
- Iteration → Scientific mindset — experiment, measure, improve
The closing message is simple and powerful: "Always look at your data." The skills audit framework and slides are available via QR code shared at the talk.
🎯 Key Takeaways
🔑 Key Takeaways
- The harness is data science — OpenAI's agent harness works because it includes logs, metrics, and traces — not just specs and unit tests
- Vibes-based evaluation is a regression — AI engineering has abandoned rigorous ML practices like data visualization, metric alignment, and human label validation
- Generic metrics are a trap — "helpfulness" and "hallucination" are too ambiguous; build bespoke failure modes from exploring your own traces
- LLM judges need validation — treat them as imperfect classifiers with train/dev/test splits, and use precision/recall for imbalanced tasks
- Synthetic data needs systematic design — identify varying dimensions, generate values per dimension, and take the cross product instead of asking an LLM for generic examples
- Binary beats scales — pass/fail is more actionable and easier to validate than 1-to-100 scores
- Domain experts must label data — developers without domain knowledge produce poor labels; never outsource understanding
- Criteria drift is real — people don't know what they want until they look at data, so rubrics must evolve continuously
- Automation has limits — LLMs can find obvious errors but miss the product-level failures that matter most to users
- Build delightful annotation interfaces — remove all friction from data review; raw JSON kills engagement
- Every dashboard metric must earn its place — uncalibrated LLM scores and meaningless similarity metrics waste attention
- Always look at your data — the single most important practice, and the one most commonly skipped
🔗 Resources & Links
- Everything shipped at Interrupt 2026 — LangChain's roundup of all announcements from the conference
- LangSmith Engine — LangChain's observability and evaluation platform for LLM applications
- LangChain — the framework and platform for building LLM-powered applications