Overview
IndyDevDan builds a Pi Agent observability system to measure the trade-off trifecta — performance, speed, and cost — across three spec formats: Markdown, HTML, and VSpec (HTML with GPT Image 2 visuals). Using three parallel Gemini 3.5 Flash Pi coding agents, he races them against the same prompt to demonstrate why observability is non-negotiable for production agents. He also demos "Steelman," a product-focused bear-thesis agent that generates generative UI components with real research data. The core message: if you don't measure your agents, you're gambling with tokens, not engineering.
1 The Core Thesis: More Useful Tokens Win
Starting from Anthropic's viral post on "the unreasonable effectiveness of HTML" and OpenAI's benchmark-gapping image model, Dan frames the question: Markdown specs, HTML specs, or something else?
The answer: more useful tokens outperforms fewer useful tokens. The key word is "useful." As you build engineering and product agents, the question isn't just "what's better" — it's "what's the trade-off between performance, speed, and cost?"
2 Agent Observability Architecture
The architecture is simple:
- Centralized server — Agents stream events to it (also persists to DB)
- UI dashboard — Reads events, displays them in multiple views
- Closed-loop feedback — Engineer ingests the information and improves the agents
Every event, every turn, every tool call comes in for every agent. "We can observe everything. And that means we can improve everything."
3 The Trade-Off Trifecta
Three axes to measure for any agent solution:
- Performance — Did it produce the right result? Did it call the right tools?
- Speed — Tokens per second, total duration, number of turns
- Cost — Total tokens consumed, dollar cost per run
Dan races three Pi coding agents with the exact same prompt but different spec formats (Markdown, HTML, VSpec) to compare these dimensions side by side.
4 Steelman: A Product-Focused Agent
Beyond engineering agents, Dan showcases "Steelman" — a bear-thesis agent that counters your investment thesis with researched arguments and generative UI.
How It Works
- Present a bull thesis (e.g., "Apple is an underappreciated AI distribution winner")
- The agent researches and generates the strongest bear case against your thesis
- Produces generative UI components: pie charts, scorecards, timelines, tables
- Cites 40+ real references from tool-calling research (not hallucinated)
Key Insight from the Demo
Mac Mini represents less than 2% of Apple's total revenue. The Claude Mac Mini craze, while viral, proves developers want decentralized local AI — but selling cheap, low-margin silicon boxes isn't great for Apple's repeat revenue model. The agent catches nuances that hype-driven analysis misses.
5 Observability Dashboard Deep Dive
The Pi observability dashboard has multiple viewing modes:
- Swim Lane View — All agents side by side with events flowing in real time
- Single View — Detailed drill-down into one agent's full run (events, cost, tokens/sec, context)
- Race Mode — Left-to-right comparison of events chunked by turn, seeing exactly when each agent does what
- Function Mode — Super compressed view for dense information
- Form Mode — Visually aesthetic view for presentations
You can click into any event to see exact tool call arguments, assistant messages, thinking traces, system prompts — everything.
Surprising Finding
The Markdown agent used more tokens (170 events, 29 turns) than the HTML agent (100 events, 17 turns) despite Markdown being more token-efficient as a format. This could be variance or could indicate the HTML spec made the agent more focused. Without observability, you'd never know.
6 Gemini 3.5 Flash Assessment
Dan's take on Gemini 3.5 Flash:
- Near state-of-the-art for the cost — "really surprised me in a lot of cases"
- Insane tokens/sec speed — "cracked" performance for product agents
- Price controversy — Came in higher than expected, "not really a Flash model anymore"
- Multimodal leader — Best choice for VSpecs with image inputs (trained on YouTube)
7 Three Spec Types Compared: Markdown vs HTML vs VSpec
Markdown Spec
Classic, tried and true. Best for preserving tokens and text-focused results. Templated sections: Task, Objective, Problem, Solution, Relevant Files.
HTML Spec
Same content as Markdown but rendered in HTML. Provides a visually richer portrayal — especially for UI components. The agent can render mockups of quote components, timeline components, valuation gauges directly in the spec. More tokens consumed, but potentially better agent focus.
VSpec (Visual Spec)
HTML spec enhanced with GPT Image 2 generated images. Each component mockup includes a visual prototype. Costs $1-3 extra for image generation (not captured in token observability), but the planning experience is "superior."
8 VSpecs: The Future of Planning
Dan's strongest position: "All my plans now are VSpecs."
- Image tokens are highly useful for multimodal models like Gemini 3.5 Flash
- You can clearly see what the model is thinking in a visual way
- HTML prototypes rendered into plans provide demo proof-of-concepts
- Works for frontend, backend, DevOps, and product work
- Build prompts now include: "if there are any images inside the plan, you must read them"
Two constraints of agentic engineering: planning and reviewing. Visual specs dramatically help with planning.
9 Tokconomics: The Agentic Value Chain
Dan's framework for thinking about agent economics — the agentic value chain:
- Step 1: Use more tokens — Token-maxing, the starting point. Great place to start, terrible place to end.
- Step 2: Generate value from tokens — Move from raw token consumption to producing valuable outputs.
- Step 3: Arbitrage your tokens — Capture revenue from the value your agents generate. This is where product agents live.
Agent observability is essential for moving up this chain — you need to understand what your agents cost to know if the value they generate exceeds that cost.
10 Composable Agent Harness: Stacking Extensions
Dan builds "slices of composable customization" via Pi coding agent extensions:
- Pi Observability Extension — Captures all events from every agent via a single extension
- Pi Communications Network — Flat agent-to-agent communication (any agent can coordinate with any other)
- Sub-agent Delegation — Via another Pi coding agent extension
- Session Digging — Capability for introspecting sessions
"I'm building a new agent harness every single week. Some are hitters, some are losers." The extensibility of the Pi agent harness enables this rapid experimentation.
🎯 Key Takeaways
🔑 Key Takeaways
- If you don't measure, you can't improve — The oldest rule of engineering applies directly to agents. Without observability, you're gambling with tokens.
- The trade-off trifecta: performance, speed, cost — Every agent decision should be evaluated on all three axes, not just "does it work?"
- More useful tokens beats fewer useful tokens — The key word is "useful." HTML and image tokens may cost more but deliver better agent focus and results.
- VSpecs (visual specs) are the future of planning — GPT Image 2 makes it possible to embed visual mockups into specs. Dan now uses VSpecs for all planning.
- HTML specs don't necessarily cost more — In testing, the Markdown agent used more tokens (170 events) than the HTML agent (100 events). Format ≠ cost.
- Product agents need observability even more than engineering agents — When agents execute thousands of times per day, understanding cost per run is critical for business viability.
- Cost per intelligence > cost per token — A more expensive model that delivers better results is worth paying for. Evaluate intelligence, not just token price.
- Gemini 3.5 Flash excels at multimodal — Best choice for VSpecs with image inputs. Near state-of-the-art performance at competitive cost.
- Tokconomics: use → generate value → capture revenue — The agentic value chain. Most people are stuck at step 1. Observability helps you reach step 3.
- Composable extensions enable rapid experimentation — Build new agent harness slices weekly. Stack winners, discard losers.
- Counterpositional agents beat sycophantic defaults — Agents naturally tell you what you want to hear. Building agents that challenge your assumptions is more valuable.
- After observability comes evals — Once you can measure, the next step is running specs over and over to find average performance. Observability is step one.
🔗 Resources & Links
- Pi Agent Observability — Open-source codebase for the observability system shown in the video
- Top 1 Opportunity for Senior Engineers — Previous video on the agentic value chain
- Pi to Pi Agent Communication — Flat agent communication network
- Claude Code Observability — Multi-agent orchestration observability