Pi Durable in Practice: Building a 24/7 AI SRE Agent

0:00Martí Blanes builds a 24/7 AI SRE agent — a software/site reliability engineer agent that investigates incidents, fixes the code, and follows the release until the service is healthy again — as a practical demonstration of Pi Durable, the durable-agent harness from Pi.

What Pi Durable gives you

0:33Long-running agents need three things most harnesses don't provide: progress saved across interruptions, the ability to wait for human approval, and resume after a restart. Pi Durable saves the conversation itself, the application state, and task checkpoints so execution can continue from a checkpoint after a restart. Blanes sees it as "a piece of that system" his community has been discussing.

The SRE agent

1:58The build has four stages: investigate (detect, triage, investigate — building evidence and a hypothesis), fix & verify (reproduce the bug, fix it, verify with regression/component tests), deploy & observe (prepare the deployment, back up for rollback, run a 10% canary, observe, then request promotion to 100%), and resolve (final verification, mark resolved, produce a markdown report). The key property: it loops — if the canary looks unhealthy, it goes back to investigating and reproducing.

The demo run

4:03A simulated incident — a cannot read properties of undefined error — is injected. The UI (built for clarity, strippable if you only want the backend) shows the reported error, stack trace, step events, evidence, and current hypothesis filling in live. The agent reproduces the failure, changes two files, verifies with regression tests, and reaches awaiting deploy approval — where a human reviews the changed files and hypothesis before approving the 10% canary. The deployment itself is simulated, but the fix, reproduction, and verification are real (a live Codex session did the work). The finished run produces a markdown report with status, cause, fix, and verification.

Under the hood

7:29The structure: bring together the model, the coding tools, and the workflow, then hand them Pi Durable persistence storage — creating a runtime that carries incidents through each stage. Reopening the same persistence store lets Pi Durable recover the conversation and resume pending work (the host still has to be running). Each incident is a persistent document holding test evidence, deployment state, and current approval together — "progress is more than a chat history." The incident workflow is defined as a durable task with named phases and a saved checkpoint, giving Pi Durable an explicit place to continue after interruption.

Real incidents

8:40To go from simulation to production: an alert source feeds incidents in (he notes people use Jev, the decision model, to watch production logs and alert on anomalies), the system verifies the error comes from an authorized source, normalizes it, and starts the workflow. Two production-grade musts he calls out: deduplicate by a stable identity (the demo supports one active incident; a real system needs many), and route independent work. The workflow only advances when the expected findings are actually present, and it uses a stable request identity so a retry refers to the same submission — guarding against context rot in long tasks. A dedicated findings tool is how the agent writes findings back into the safe state, and test evidence comes from real execution: a setup error does not qualify as reproducing the bug.

Human approval

14:43The approval API binds each approval to an incident plus a deployment cycle, action, and candidate — and saves the decision before continuing. Two properties worth copying: stale requests are rejected with a conflict, and retrying an already-accepted request has no extra effect (idempotent). While waiting, the durable task waits without asking the model for more turns. The separation of concerns he highlights: the application layer defines the policy; Pi Durable preserves the work and the decision. And the reason the reproduce step is non-negotiable: if you can reproduce the error, you can verify the fix with the exact same flow (browser QA, Playwright) after deploy.

Claims checked

Sources & further reading