Memory-Aware Agent Harnesses for Anti-Money Laundering

0:12A workshop from AI Engineer NYC, not a product pitch: Nacho Martinez (AI developer advocate at Oracle) walks through how to build a memory-aware agent harness for a real anti-money-laundering workload, then hands everyone a codespace to build it themselves. The throughline: stop obsessing over frontier models — the harness is "the interesting part, and the thing you can actually customize."

The AML premise

4:57The working example is a medium-sized bank with streams of transactions that need autonomous triage. Five categories of suspicion — e.g. a $1M deposit followed by three days of $50k ATM withdrawals. The point is to flag cases so a human analyst spends less time hunting. The deeper framing: OpenAI launches "10,000 agents" to solve a math problem, but those agents only work autonomously because someone built the harness around them — the context window is limited, and memory is something an LLM doesn't have inherently, so "we need to implement these ourselves" to make agents run 10 or 100 hours autonomously.

The agent stack & the data layer

5:54When you talk to an AI agent you're at the application layer (chatbot, interface); below that are the model (the reasoning core you get "no say in"), the infrastructure, and compute. The workshop focuses on the data layer, because "everything we can manipulate and work around is on this part" — especially when consuming a third-party model you don't own. Inside the data layer sit the agent loop, a gateway + MCP for external sources, the memory layer, context engineering (summarize/compact/shrink the working window), and tools/skills — all in service of keeping the context window small.

What a harness is

9:17An agent, by his human analogy: powered by an LLM (the brain), augmented by a database for memory, extended through tools for action (the extremities), grounded in inputs (the senses) — and with limited attention, because "we can't watch seven movies at the same time." The reasoning is provided by the frontier labs and you don't control it; what you do control is memory, tool use, and perception — and the harness is "all the scaffolding you need to build those three components." His claim: the harness is the most interesting topic in AI right now, because the same frontier model "performs better or worse" for a given task purely on harness configuration — "proven" by just changing the harness.

Memory: short vs long-term

15:58Short-term memory is the context window itself — anything that vanishes when the conversation ends. Long-term memory he splits into three proposed categories: episodic (past conversations, tool logs, summaries), procedural (skills and workflows — sequences of skills), and semantic (who you are, your preferences, favorite framework — the "unspoken" stuff, like the fact that nobody is singing in this lecture hall right now). You can't just dump everything into short-term memory, not only because it's limited, but because "the bigger the attention matrix, the worse it performs."

Context engineering & the KV cache

18:00The evidence he cites: a problem solved at zero tokens is solved far more accurately than the same problem at 256k tokens — and the limit is now a million, "because of the KV cache." Even if the advertised limit says 256k or 1M, "if you keep going, performance goes horribly." Two countermeasures: context summarization (ask for a short summary and start the next turn from it) and compaction (chunk the conversation into a database or file with an ID, and pull by ID + offset when needed — "kind of like reading from a file").

Files vs databases

13:19He calls out the "files vs databases" war directly. Multiple agents writing to one file hit consistency problems — which write came first, what happens to a reader who read before a write — and today's workaround is git worktrees (an audience member nails the answer). His point: that's a workaround, and the underlying problem was "already solved by databases 30 years ago" via transactional / ACID consistency (atomic, consistent, isolated, durable writes). Files are great for unstructured data; databases for transactional use cases — you don't have to pick one.

The agent loop

19:46The agent loop is the driver. It runs a context assembly step that pulls from skills, tools, previous conversation, files, and the database — dynamically — then reasons (an API call to the model), then acts, "automatically and infinitely if you want." In the workshop it's bounded: max 8 iterations and 60 seconds of reasoning each (configurable; he just doesn't want to burn OpenAI credits). Every part of the loop is something "we, as engineers, get a say in."

Toolbox, skillbox & distillation

20:53The problem with hundreds of skills: each is 3–5k tokens, so loading them all blows the context instantly. His fix — the toolbox/skillbox pattern: vectorize the tools and skills into embeddings, and when the agent loop needs something, do a vector-similarity search, pull the top-k most relevant, solve, then offload. In the workshop, k=3 against one vector store for tools and another for skills, optionally refined by a reranker (embedding model + reranker to pick the single best of three). 22:30Skill distillation is the learning loop: take previous conversations and runs, extract something meaningful, and write it back as a SKILL.md into the same skillbox — he names Hermes Agent as "one of the innovators" in creating skills that capture preferences (coloring style, etc.) that the vector search then surfaces automatically.

Continual learning

24:12How does a "frozen" model get better? You don't control the frontier weights, but you control the data layer. Three categories of continual learning: parametric memory (modifies weights — LoRA, fine-tuning), representational memory (embeddings), and nonparametric memory (everything in this talk — the memory layer itself). The workshop focuses on the third; the other two are covered in his DeepLearning.AI course.

Instruction hierarchy

25:59Every chat-completions call carries an implicit weight hierarchy. Most important is the system prompt, then developer/app instructions, then the user request, and tools and skills carry the least weight — a "plug-and-play place you can pull things from." That's a feature, not a bug: it lets the model focus on some things more than others, and it's why you shouldn't stuff everything into the system prompt.

Semantic cache & cost

36:25A practical economics point: in a typical request, tools, skills, and retrieved docs are ~80% of the payload (35k–100k tokens), versus a ~100-token user query. The reason you aren't charged 5× more is the semantic cache: if 80% of the request matches the previous one, it isn't recomputed. It's still expensive — just 4–5× cheaper than it would be without that caching. He also cites a paper comparing harnesses by initial system-prompt size: Pi is the most minimal (~6,000 tokens) vs others at 30–40k tokens.

The AML application

27:05The demo is an autonomous AML agent running 24/7 on Oracle Cloud Infrastructure (a Grok model on OCI; fake data). It streams transactions, flags suspicious ones, searches knowledge spaces built as vector stores, runs tool calls, and drafts suspicious-activity reports — crucially, it does not auto-block accounts ("that would ruin a person's life"); it prepares decisions and prints them for a human to inspect, saving analyst time during discovery. The end state of the workshop: an agent connected to a real database doing autonomous analysis and decision-making that a human then consumes.

Q&A nuggets

40:49Keep the memory layer simple: start with just short-term + long-term; promote anything worth keeping from the context window into long-term memory as raw text, then vectorize it later and let the agent loop recall it (a "recall" step between observing and reasoning that pulls relevant memories by vector distance). 43:16MCP's real value for databases (from a self-described "database guy"): pre-create proven SQL queries and let agents call those, instead of hallucinating a fresh SQL query every time — SQL has ~15 engines with dialect-specific hints and subtleties. 44:24Embeddings beat regex: keyword/word search over skill text doesn't scale (regex matching degrades as files grow), while precomputed embeddings capture hidden meaning in 384–1000+ dimensions, and hybrid search — semantic + keyword, fused via RRF — outperforms either alone. 47:53Build custom harnesses for specific problems — and "start from scratch, because that way you can strip 90% of what you don't need"; you don't need to re-create Claude Code, you need a small harness that attaches to a specific use case.

Claims checked

Sources & further reading