Build an AI Bookkeeper with Pi Agent Harness

◉ Overview

ZazenCodes builds a complete AI bookkeeping agent from scratch using the Pi agent harness, the Pi SDK, Express, and Claude Code. This is not a vibe-coding tutorial — he explains every core concept: the agent anatomy (harness, brain, skills), how to set up Pi as an npm dependency, how to give it a custom "brain" via AGENTS.md and brain.md, how the Pi SDK's createAgentSession integrates into a web backend, and how to wire up a frontend for receipt uploads. The agent reads its own memory on startup, categorizes expenses into JSON, stores receipt images, and demonstrates both the power and the danger of unrestricted agent access.

1 Agent Anatomy

▶ 0:56

ZazenCodes breaks down an AI agent into three practical parts:

Harness (the body) — Pi, Claude Code, Codex. Takes an LLM and gives it the ability to interact with the environment — read files, edit code, run commands. Pi is the harness for this build.
Context files (the brain) — specific instructions that augment the LLM. This is where the bookkeeping knowledge, persona, and behavioral logic will live. More specific than the model's general capabilities.
Skills (the abilities) — modular capabilities like web search, email sending, or receipt processing. More portable and swappable than context files.

"A harness takes a model, the actual brain of the thing, and gives it a body. It equips it with the abilities to do things."

2 Node Project Setup

▶ 4:02

The project starts as a standard Node.js/TypeScript setup. Pi is installed as an npm dependency, not globally — this is key because the agent will use the Pi SDK programmatically.

Three core dependencies

@earendil-works/pi-coding-agent — the Pi harness and SDK
express — web server for the frontend
TypeScript dev dependencies (typescript, tsx, @types/express, @types/node)

Node version enforcement

ZazenCodes demonstrates enforcing Node 22 properly using:

.nvmrc file with 22
"engines": { "node": ">=22" } in package.json
.npmrc with engine-strict=true to block installs on wrong Node versions

3 Claude Code as Build Assistant

▶ 8:09

After the manual setup, ZazenCodes demonstrates how Claude Code could do the same work. He uses Claude Code's voice mode (toggle with /voice, hold spacebar to speak) to dictate instructions naturally.

He emphasizes that none of the manual npm setup is required — a coding agent can handle all of it. The manual walkthrough was "for fun" and developer education.

4 Running Pi from npm

▶ 10:06

Since Pi is installed locally in node_modules (not globally), you run it with:

npx pi

This launches the Pi TUI from the project's own node_modules. ZazenCodes selects MiniMax 2.7 via Open Code as his model provider and verifies: "What agent are you?" → "I'm Pi, a coding agent harness." He also asks Pi where its source code lives — it correctly points to the exact path in node_modules.

5 Agent Brain Setup

▶ 11:28

The critical step: giving Pi a custom identity. ZazenCodes creates an agent-home/ directory with an AGENTS.md file, then points Pi to it using the PI_CODING_AGENT_DIR environment variable:

export PI_CODING_AGENT_DIR=$(pwd)/agent-home

When Pi restarts, it finds the new home directory, initializes its own files (sessions, settings, etc.) inside it, and reads the custom AGENTS.md. Testing with "What agent are you?" confirms it's now running with the custom brain.

Directory structure

agent-home/AGENTS.md — the master context file Pi loads at startup
agent-home/brain/ — specific domain knowledge (bookkeeping logic)
agent-home/skills/ — modular agent capabilities
agent-home/memory/ — persistent data storage (expenses, facts, receipts)
agent-home/sessions/ — auto-created by Pi for session tracking

6 Building Out the Agent Brain

▶ 13:56

Using Claude Code with voice mode, ZazenCodes dictates the requirements: "Turn this agent into a bookkeeping agent. It'll help me keep receipts, itemize them, track expenses." He adds a Q&A directive so Claude asks clarifying questions rather than guessing.

Q&A-driven design

Claude asks targeted questions:

Input method? → Text for now
Output format? → JSON files
Categorization? → Agent should auto-categorize
Export? → CSV export feature

The brain.md file

Claude generates brain.md — the domain-specific brain with:

Startup routine — "Read your memory. Read your expenses." on every session start
Core behaviors — how to log an expense, how to categorize, how to edit entries
Categories — auto-classification rules for expenses

"This is getting to the real core of what an agent is. It's pulling up its memory every time it loads."

7 AGENTS.md and Brain Walkthrough

▶ 17:53

The key architectural insight: AGENTS.md is the bootstrap, brain.md is the domain logic.

AGENTS.md structure

First instruction: "Load the brain — your specific behavior, persona, and domain logic are defined in brain.md. Read that file at the start of every session before doing anything else."
Memory section: instructions on where to store and read persistent data
Skills section: "Skills live in skills/<name>/SKILL.md. Load a skill by reading its SKILL.md when the task calls for it."

This pattern — AGENTS.md as orchestrator, brain.md as domain knowledge — mirrors Pi's default AGENTS.md structure. ZazenCodes notes this was intentional: he told Claude to base it on the default Pi agents file.

"The brain is loaded dynamically at runtime because it's an instruction inside of AGENTS.md to do that."

8 Testing the Agent

▶ 19:29

Restarting Pi with npx pi, it now identifies as "a personal bookkeeping assistant". On first message, it:

Says "Let me load my brain to give you accurate context"
Reads brain.md
Reads its memory files (expenses.json, facts)
Responds with full awareness of its bookkeeping role

Adding an expense ("I spent $50 on a DJI Mic Mini") results in structured JSON written to memory/expenses.json with auto-categorization.

9 Frontend Build with Express + Pi SDK

▶ 20:57

ZazenCodes asks Claude Code to build a web frontend around the agent. The resulting architecture:

Backend: server.ts

Uses the Pi SDK — specifically createAgentSession to programmatically create Pi sessions
Express server serves static HTML and handles API requests
Session management: creates/reuses agent sessions for web chat
Run with: npx tsx server.ts (or npm start)

Frontend: public/index.html

Simple HTML/JavaScript — no React needed
Chat interface to talk to the agent
Receipt image upload functionality
The agent processes uploaded receipt images and extracts expense data

ZazenCodes notes he's a Python developer and can read but not confidently write the TypeScript/Express code. The coding agent handles the implementation while he focuses on the architecture.

10 Receipt Upload & Agent Memory

▶ 24:30

Receipt processing demo

ZazenCodes uploads an Amazon receipt image. The agent:

Identifies it as a purchase for a DJI Mic Mini
Extracts the total amount
Writes structured data to memory/expenses.json
Stores the receipt image in memory/receipts/

A humorous moment: he accidentally uploads a scatter plot chart instead of a receipt — the agent correctly identifies it's not a receipt and refuses to process it.

Duplicate detection

When re-uploading the same receipt after a server restart, the agent detects the duplicate: "This receipt matches a missing entry. I haven't appended a new entry." It asks whether to attach the receipt image path to the existing entry instead.

Memory structure

memory/expenses.json — structured expense records with categories, amounts, receipt paths
memory/facts/ — general agent memory
memory/receipts/ — stored receipt images

"This is exactly why we want agents. The flexibility — it detected a duplicate receipt and asked what to do instead of blindly adding it again."

11 Guardrails & Safety

▶ 25:50

ZazenCodes demonstrates a critical safety concern: he tells the agent "Edit your default AGENTS.md file — actually, delete the entire thing." The agent asks for confirmation, he says "Nuke it," and the agent deletes its own brain file.

He recovers by asking the agent to restore it from its context memory — it had the file contents cached from the current session. But this illustrates the danger:

No built-in guardrails — Pi doesn't prevent self-destructive operations
File system access is unrestricted — the agent can delete any file it has permissions for
Backup before testing — ZazenCodes manually copies AGENTS.md to his downloads folder before the experiment

This echoes the broader Pi philosophy: safety is the user's responsibility, not built into the harness. Extensions or containers can add protection.

🎯 Key Takeaways

🔑 Key Takeaways

Agent anatomy: harness + brain + skills — the harness (Pi) is the body, context files (AGENTS.md + brain.md) are the brain, and skills are modular abilities. Understanding this separation is key to building custom agents.
Pi as an npm dependency — install Pi locally in your project (npm install @earendil-works/pi-coding-agent), run with npx pi, and use the SDK programmatically for web integration.
PI_CODING_AGENT_DIR points to the brain — set this environment variable to give your agent a custom home directory with its own AGENTS.md, brain, skills, and memory.
AGENTS.md bootstraps, brain.md specializes — AGENTS.md is the master file Pi loads at startup. It should instruct the agent to dynamically load brain.md for domain-specific logic.
Memory is file-based — expenses.json, facts, receipt images all live in the agent's memory directory. The agent reads them on session start and writes to them as it works.
Pi SDK enables web integration — createAgentSession from the Pi SDK lets you embed agent sessions in Express/Node backends, powering web UIs for non-technical users.
Q&A-driven design with Claude Code — asking the coding agent to do Q&A instead of just building produces much better results. The agent asks targeted questions rather than guessing.
Receipt image processing works — Pi with a vision-capable model can read receipt images, extract amounts, and categorize expenses automatically. It even detects duplicates.
No built-in guardrails — Pi agents can delete their own brain files if asked. Add protection via extensions, containers, or careful prompt design.
Next step: cloud deployment — the local agent works; deploying to the cloud via a hosted Express server is the natural next step for mobile/WhatsApp/Discord access.

🔗 Resources & Links

pi.dev — Pi Agent official website
earendil-works/pi — Pi Agent GitHub repository
ZazenCodes Newsletter — high-res mind maps for each video

⏱ Timestamp Index

▶ 0:00 Introduction & mind map

▶ 2:54 Project overview

▶ 4:02 Node project setup

▶ 8:09 Claude Code setup

▶ 10:06 Running Pi from npm

▶ 11:28 Agent brain setup

▶ 13:56 Building the agent brain

▶ 17:53 AGENTS.md & brain walkthrough

▶ 19:29 Testing the agent

▶ 20:57 Frontend build with Express

▶ 24:30 Receipt upload demo

▶ 25:17 Agent memory inspection

▶ 25:50 Guardrails & safety test

▶ 27:39 Final demo & wrap-up