🎯 A New Class: Automation Models 0:00
The video opens by naming a category that's emerging fast: automation foundation models — models that produce only structured outputs, extremely fast, without the token-by-token text generation of an LLM. The structured output can be translated directly into actions (the host's example is a model playing Doom), and the canonical closed-source example is Jev from TypeSafe, which we've covered separately. Now open-source alternatives are appearing, and the one under review is Needle 3 🔗 — an automation model "for tiny devices" that's 53 million parameters and 35 MB, small enough to run on a phone.
🏠 The 66ms Home Automation Demo 0:57
The demo is a living room with several appliances, fed a deliberately messy, multi-intent voice query: "I'm feeling really hot today. Can you turn on the fan? Turn down the temperature to 10°C, and also turn on the bedroom light." The model extracts all three intents from the unstructured natural language and produces the correct function calls — the temperature set to exactly 10°C, the fan on — in about 66 milliseconds.
The thing to focus on isn't the speed but the shape of the output: it looks like a function call — the model generating a JSON schema directly from the unstructured input, without an intermediate text step. That's the whole category in one screenshot: "unstructured data in, structured action out."
⚖️ Why Not Just an LLM? 2:37
The obvious objection — "an LLM can already do function calling" — is true but misses the economics. An LLM generates text one token at a time, emits a text description, and then needs a parser to turn that into a JSON schema. Needle 3 is "specifically designed to generate these structured outputs directly," which means no parser on top and, critically, no hallucination — it can't invent a value that isn't in the schema, because it validates the schema and expected inputs directly.
| LLM + function calling | Needle 3 | |
|---|---|---|
| Output | Token-by-token text + description | Structured schema directly |
| Parser required | Yes | No |
| Hallucination | Possible | Not possible (schema-validated) |
| Size | 7B ≈ 4GB (4-bit) | 35MB (smaller than Whisper small) |
| Speed | Seconds (reasoning) | 2 calls < 97ms on CPU |
It's the third model from Cactus, whose whole focus is "powerful but tiny models for automation and function calling." The comparison point is explicit: where Jev is a large System-1 classification model in the cloud, Needle 3 is the same idea compressed onto-device.
🏗️ Architecture: Simple Attention Network & Slicing 3:42
The architecture is what Cactus calls a "Simple Attention Network" (SAN). Start with a standard transformer and remove the feed-forward network, replacing it with a multi-layer perceptron of just 225.6 thousand parameters, plus an n-gram lookup that can produce an embedding for direct lookups. The result is "substantially smaller compared to a standard transformer."
The limitation is explicit and deliberate: it "cannot do anything else" — ask it "what's the capital of France?" and it returns an empty response, because there's no tool definition for that. "This limitation is a feature, not a bug." But the flip side is that a tiny model can be confused by instruction phrasing, which the gotchas section makes concrete.
💻 Function Calling in Code 6:40
The developer experience is a plain Python function wrapped in a decorator: you define the function, its typed inputs, and a docstring — "the description the model uses for selecting a specific function" — and the decorator registers it. Two functions in the demo (set_lights, lock_doors) plus an output schema (rooms, status, brightness; locked/unlocked, which door) are all it takes.
Feed it "Dim the living room lights to 30 and lock the front door" and it decomposes the raw text into the correct structured JSON for both calls — selecting "living room," setting brightness to 30, and marking the front door locked. It also returns a confidence score, which the host is careful to distinguish from Jev's calibrated probability: this is "how confident the model is about a specific decision," a signal to threshold on rather than a true probability. Decode speed is "pretty good given it's a CPU on a Google Colab notebook."
⚠️ Four Gotchas 7:54
The most valuable part of the video is the failure modes — "knowing where a model breaks matters more than knowing where it shines." Four concrete gotchas from actually testing it:
| # | Gotcha | What happens | Fix |
|---|---|---|---|
| 1 | History leakage | It keeps conversation history; multi-turn queries without a reset produce wrong calls (e.g. "order me a pizza" with no pizza tool → an inaccurate schema with low confidence) | Reset state between calls |
| 2 | Missing default values | Correctly-called functions with no defaults return empty/no results | Always include good defaults in function definitions |
| 3 | Messy input | Structured extraction works on clean descriptions (an invoice); message-like text → none results | Be careful what text you feed it |
| 4 | Classification limits | Direct intent classification is weaker than a System-1 model like Jev | Don't use it for classification; use Jev/System-1 for that |
Gotchas 1 and 2 are the ones that will bite a real deployment: they're not "model quality" issues but state management issues — and they're exactly the kind of thing you'd only discover by running the model, not by reading the README.
🔍 Embeddings, Slicing & Verdict 11:18
Two findings round out the picture. First, a surprise: Needle 3 can generate embeddings, and the host tests whether they're good enough for direct cosine similarity on a small corpus — "a decent job, 4/6, 5/6 mean-centered" — suggesting real potential for on-device semantic search/retrieval without a separate embedding model.
Second, the slicing story is the deployment story: you can cut the model to the exact size your hardware allows, trading capability for footprint. His takeaway is a prediction and a posture: this is "great for on-device automation" and we'll see more of it, and he's already planning to fold it into his own local automation because "it's completely local and on-device."
💡 Key Takeaways
- Automation models are a distinct category. Structured-output-only, fast, no token-by-token text — Jev in the cloud, Needle 3 on-device.
- 66ms function calling on a CPU, no parser, no hallucination. Raw unstructured text in, validated JSON schema out.
- The architecture is a "Simple Attention Network." Transformer minus the feed-forward, plus a 225.6k-param MLP and an n-gram lookup — and 20 sliceable layers.
- Slicing is the deployment superpower. Any subset of layers is a valid smaller model: 2 layers ≈ 13MB for phones, 20 layers = 35MB.
- Limitation is a feature. "Capital of France?" → empty response, because there's no tool for it. It won't pretend.
- Four gotchas to design around: reset state between calls (history leakage), always provide function defaults, feed it clean text, and don't use it for intent classification.
- Bonus: it can do embeddings. Decent cosine-similarity performance — potential on-device semantic search.
🔗 Resources & Links
- 📦 github.com/cactus-compute/needle — the repo
- 🤗 huggingface.co/Cactus-Compute/needle3 — the model card
- 📰 cactuscompute.com/needle — the Cactus blog post
Source video: youtube.com/watch?v=qbN559fQn7k — the Colab notebook is linked in the video description.