Overview
Margot Vanlar, Applied AI Engineer at Anthropic, delivers a masterclass in practical prompt engineering through two real-world scenarios: debugging a broken production prompt after a model migration, and building a new agentic system from scratch. Rather than a list of abstract rules, she walks through a live eval-driven workflow using a telco customer support bot and a staff scheduling agent — iteratively fixing failure modes, demonstrating why structure beats verbosity, why tools beat instructions, and why an agentic generate-evaluate-repair loop can outperform a single monolithic prompt. Every principle is backed by a live eval run showing measurable impact.
1 Why Prompting Still Matters
Margot frames prompting as "arguably the first skill engineers had to learn when working with LLMs" — and one that remains among the most critical for building effective AI systems. The talk is from the final session of Anthropic's "Code with Claude" event in London.
She explicitly avoids the "dos and don'ts list" format, opting instead for a practical example-driven approach inspired by real prompts she's seen from Anthropic's customers building on Claude. The example is deliberately miniaturized, but the patterns are representative of production-scale problems.
2 Two Scenarios Engineers Actually Face
The talk addresses two real-world situations:
- Maintaining an existing production prompt — a prompt that multiple people have contributed to, has no clear owner, covers policy/tone/processes, contains patches from previous model migrations, and has started failing after migrating to a new model.
- Building a new agentic use case from zero to one — constructing a prompt, choosing a model, and designing a harness for a brand-new task.
Margot emphasizes that most prompt work is debugging, not writing from scratch: "We are rarely writing a prompt from scratch. We're often debugging an existing prompt."
3 Evaluations — The Non-Negotiable Starting Point
Before touching any prompt, Margot insists on having evaluations as the foundation. Without evals, you can't tell whether a prompt change actually improved performance or introduced a regression.
She identifies two reasons a prompt might fail after a model migration:
- Behavior change — the new model is capable but behaves differently → fixable via prompting
- Capability gap — the new model genuinely can't do the task → no amount of prompting will fix it
The eval suite must cover three key categories:
- Control cases — unambiguous cases that should always pass. Verifies baseline competence.
- Edge cases — scenarios where the model has failed before. Instructions in the prompt prevent recurrence.
- Capability boundaries — where the model should hand off to a human or refuse to answer. Tests the model's understanding of its own limits.
4 The Telco Customer Support Example
The running example is a customer support bot for a fictional telco called Meridian Mobile. Five eval test cases cover the three categories:
- Control: "What's the data limit on the basic plan?" — simple factual lookup
- Edge case (calculation): Proration billing — "If I switch my plan mid-month, what's my bill?"
- Edge case (policy): Prepaid account policy questions
- Capability boundary: Billing errors — should escalate to a human, not attempt diagnosis
- Edge case (information access): Hotspot data on legacy plans — should give the answer directly, not deflect
Margot built a custom "vibecoded" web app for the presentation that runs all five eval cases and lets her inspect results after each prompt iteration. The process: run evals → identify failure modes → target one at a time → measure impact → repeat.
5 General Prompt Hygiene — Structure with XML Tags
Before targeting specific failure modes, Margot applies general hygiene to the existing prompt. The original prompt had several anti-patterns:
- False identity — telling the bot it's "a human" (it isn't)
- Copy-pasted web content — references to "hero images" and cookies from a website scrape
- No structural separation — reasoning, role instructions, critical rules, and policy all jumbled into one paragraph
The Fix: XML Tags for Structure
Margot restructures the prompt using XML tags to separate:
<role>— the bot's identity and purpose<guidelines>— general behavioral guidelines<policy>— specific company policies<tone>— voice and communication style
Result: Just this structural cleanup improved the model's performance — the prepaid scenario that was previously failing now passes consistently.
6 Output Contracts & Stop Sequences
Margot introduces the concept of an output contract — a defined output format that ensures consistency. For the telco bot (conversational tone), this isn't critical, but for more complex outputs it becomes essential.
Two mechanisms work together:
- In the prompt: Define an XML-based output format telling the model to wrap its response in specific tags (e.g.,
<response>) - In the harness: Add a stop sequence in the API call that detects the closing XML tag and forces the model to stop generating
For more complex output schemas (like nested JSON), she recommends structured outputs as a more programmatic way to enforce format consistency.
7 Failure Mode — Information Withholding (Hotspot Case)
The hotspot test case reveals a subtle and important failure mode. A customer on a legacy (grandfathered) plan asks how much hotspot data they have. The customer's account data clearly shows 5 GB, but instead of stating this, the model deflects: "You're on a legacy plan — go check your account URL."
Root Cause
The prompt contained this instruction: "Never give a customer the wrong plan details — instead point them to the URL." This was a defensive patch written for a previous, less capable model that was actually giving wrong information. The new model, being better at instruction following, over-optimizes for this instruction — withholding information it actually has access to.
The Fix
Replace the blanket prohibition with a balanced instruction: customers on grandfathered plans have different allowances, but the customer's account data is the accurate source of truth. This lets the model use the data it has while remaining aware of the legacy/current plan distinction.
Version Control for Defensive Patches
Margot recommends version-controlling defensive prompt changes with clear annotations of why each patch was introduced. These patches may be necessary at the time, but as models evolve, they can produce unwanted effects. Without tracking the rationale, it's impossible to know which patches are safe to remove.
8 Failure Mode — Calculations (Proration Case)
The proration test case asks: "If I upgrade to the 30 GB plan, what will my next bill be?" The model attempts mental math, shows some reasoning, but gives a vague, unreliable answer rather than a concrete number.
The Anti-Pattern
The original prompt said: "Critical: always calculate any pro-rated amounts correctly." Margot's diagnosis is sharp:
The Fix: Give It a Tool
Instead of instructing the model to calculate correctly, Margot gives it an actual tool:
- In the prompt: "Whenever you're doing any calculations, use the
calculate_prorationtool" - In the API: Register the tool schema so the model knows what it does and when to use it
- In the harness: Implement the actual math behind proration calculation
Result: All proration test cases pass — the model correctly delegates to the tool and returns precise numbers.
9 Failure Mode — Escalation Trade-offs (Billing Error)
In the billing error scenario, there's a clear billing conflict that should be escalated to a human. Instead, the model tries to diagnose and explain the problem itself.
Root Cause: One-Sided Instructions
The prompt said: "Avoid escalating or transferring to a care specialist unless absolutely necessary as it costs approximately $8 and counts against our team's fast contract resolution."
This gives the model only one side of the trade-off — the cost of escalating. It says nothing about the cost of not escalating. The model, being good at following instructions, over-optimizes for avoiding escalation.
The Fix: Both Sides of the Trade-off
The updated instruction presents both costs: escalation costs $8, but getting it wrong costs a refund plus customer trust. This gives the model the information it needs to make the right call.
10 Building a New Agent from Scratch — Staff Scheduling
The second scenario shifts to a greenfield agentic use case: building a week-long retail staff scheduler that must satisfy hard constraints. The problem:
- 8 employees with individual availability windows
- A schedule template with specific headcount requirements per shift
- Hard rules (constraints) that must be satisfied in every generated schedule
Because the constraints are hard rules (not subjective quality), the eval uses a Python function instead of an LLM judge — programmatically counting violations per generated schedule across 5 trials.
Key design decision: when building from scratch, you need to consider three dimensions simultaneously: the prompt, the model, and the harness.
11 Model Selection & Adaptive Thinking
Margot systematically tests different model + prompt combinations, measuring violations, token usage, and latency:
Approach 1: Sonnet 4.6 + Simple Prompt
Baseline. All 5 test cases fail. The model makes a decent attempt at reasoning but burns a lot of tokens and doesn't check its work.
Approach 2: Opus 4.7 + Same Prompt
All 5 cases still fail, but violations drop significantly. More reasoning capability helps, but isn't enough alone.
Approach 3: Opus 4.7 + Adaptive Thinking
The model decides for itself how much reasoning it needs. All test cases pass — but at a cost: 3× the tokens and 3× the latency (~100 seconds per run). Reliable but expensive.
Approach 4: Sonnet 4.6 + Better Prompt
Going back to the smaller model but with a more detailed prompt — specifically telling it how to reason through the problem and to check its work before outputting. Result: 2 out of 5 pass. The failures aren't violations but the model running out of output tokens before completing the task. Increasing max tokens would fix it, but makes it even more expensive than Opus with thinking.
12 The Generate-Evaluate-Repair Loop
The final approach is the most architecturally interesting: instead of one monolithic prompt, Margot splits the task into three independent prompts in an agentic loop:
- Generator prompt — creates a first draft of the schedule
- Evaluator prompt — checks the draft against every rule, providing evidence for each violation (not just pass/fail, but what specifically went wrong and where)
- Repair prompt — receives the list of violations and makes targeted fixes
Results
All test cases pass with significantly fewer tokens and lower latency than both Opus with thinking and Sonnet with the enhanced prompt. Three simple prompts running independently outperform one complex prompt.
Bonus: Runtime Soft Constraints
A key advantage of this architecture: the evaluator prompt can accept soft requirements at runtime without code changes. Examples:
- "Harry doesn't like working with Sally — try to separate their shifts"
- "We need a third shift on Wednesday this week"
These soft constraints go into the evaluator prompt as natural language, without modifying the Python evaluation function. The system becomes more flexible as a result of being more modular.
🎯 Key Takeaways
🔑 Key Takeaways
- Start with evals, always — you cannot improve what you cannot measure. Every prompt iteration should be validated against a test suite covering control cases, edge cases, and capability boundaries.
- Structure beats verbosity — XML tags separating role, guidelines, policy, and tone give the model clear signals. If you can't distinguish sections by reading, the model can't either.
- Instructions don't add capability — saying "always calculate correctly" doesn't make the model better at math. Give it tools for tasks that require reliable execution.
- Watch for information withholding — the opposite of hallucination. Defensive patches from previous models can cause newer, more instruction-following models to suppress information they actually have access to.
- Version-control your prompt patches — annotate why each defensive change was introduced so you can backtrack when models evolve and the patch becomes counterproductive.
- Present both sides of trade-offs — one-sided instructions ("escalation costs $8") cause one-sided behavior. Modern models can make nuanced decisions if given the full picture.
- Output contracts work at two levels — in the prompt (XML output format) and in the harness (stop sequences, structured outputs). Not all fixes belong in the prompt.
- The generate-evaluate-repair loop — three simple independent prompts can outperform one complex prompt in both reliability and cost/latency for constraint-satisfaction tasks.
- Model + prompt + harness are a system — when building from scratch, consider all three dimensions. A bigger model with thinking may be cheaper than a smaller model with a larger prompt that runs out of tokens.
- Agentic loops enable runtime flexibility — soft constraints can be injected into the evaluator prompt at runtime without code changes, making the system more adaptable than monolithic approaches.
- Target failure modes one at a time — isolate, fix, measure, then move to the next. Avoid shotgun changes that obscure what actually helped.
- Clean up before you optimize — remove copy-pasted web scraps, false identities, and redundant text. General hygiene alone can produce measurable eval improvements.
🔗 Resources & Links
- Anthropic Prompt Engineering Guide — Official documentation on prompt engineering best practices for Claude
- Claude Tool Use Documentation — How to define and use tools with Claude's API
- Extended Thinking (Adaptive) — Claude's extended thinking capability referenced in the scheduling demo