Needle 3: The 8MB On-Device Model That Beats 10× Larger Models

🎯 The 8MB Model Nobody's Talking About 0:00

A new AI model small enough to fit inside a single app icon — as small as 8MB on the low end, 29MB on the high end — that runs completely on your phone, smartwatch, smart home hub, or even a tiny robot, with no internet connection required at any point. That's Needle 3, and its makers claim it beats models 10× its size at real-world tasks.

The company behind it is Cactus Compute, a small San Francisco startup backed by Y Combinator, quietly building open-source AI for mobile and edge devices. Their pitch is simple: cloud models are powerful, but they need a network, cost money every call, and send your data elsewhere. Cactus builds models and an inference engine designed to skip all of that and run the intelligence right on the hardware in your hand or on your wrist.

📜 Needle's History: Distilled Gemini → Needle 3 0:32

Needle 3 is the third version in a line, and the progression shows the design philosophy tightening over time:

The through-line is honesty about scope: Cactus never pretended Needle was a generalist. Each release does one thing well — "figuring out which function to call and filling in the right arguments" — and Needle 3 is where that specialist gets genuinely clever under the hood.

🚀 Traction & the Pebble Partnership 3:05

The signals that this is a serious, maintained project rather than a one-off experiment:

• GitHub: ~11,300 stars, 700+ forks, dozens of active contributors.
• License: recently moved from MIT to Apache 2.0.
• Activity: fixes to the engine and internal-state handling landing within the last day or two.

The stronger signal is a real hardware partnership. Pebble — the smartwatch company, whose screenless Pebble Index Ring has no display at all — runs Cactus Needle inside their app. Founder Eric Migicovsky's reasoning is the perfect illustration of the use case: because the ring has no screen, when you speak a command, that action has to happen every single time, whether or not there's connectivity. Running the model locally means a tiny footprint with performance that, in his words, "has never let them down."

🛠️ Three Jobs in One Tiny Binary 4:31

Needle 3 is built for three specific jobs, all running entirely on-device — nothing sent to a server, nothing waiting on a network:

The "dim the bedroom and lock up" example makes it concrete: one sentence becomes two ordered function calls, executed instantly with no round trip to a hub. These three jobs are normally split across separate specialized models — or dumped on a large cloud model that you "hope gets the structure right." Needle 3's pitch is one small model doing all three locally, removing a meaningful chunk of complexity for on-device apps.

🪜 Architecture: The Intelligence Ladder 7:36

The defining idea is the "intelligence ladder." A smartwatch, a Raspberry Pi, and a flagship phone want the same behavior at very different sizes — and the usual answer (train a separate model per size) means separate training and fine-tuning runs every time something changes. Needle 3 instead trained one set of weights where every depth from 2 layers up to 20 layers is, on its own, a complete and usable model, each genuinely more capable than the one below it. Pick a 2-layer version for a smartwatch or the full 20-layer for a more capable device — all from one training process, and each depth independently fine-tunable afterward. In real numbers: 4 layers ≈ 8MB, 20 layers ≈ 29MB, with working models at every depth between.

The architectural choices that make this feasible:

• Monarch Hadamard MLP replaces the standard feed-forward layer — computationally cheaper.
• Grouped-query attention with rotary position embeddings, plus causal conv taps.
• Engram n-gram memory: a hash-memory lookup where a large share of parameters are stored as memorized patterns rather than calculated live.
• Multi-lane hyper-connections: multiple parallel information pathways through the network.

The engram is the interesting bit: because "most of its parameters sit in the engram," a 121M-parameter model does the raw arithmetic of a 50M-parameter one. Cactus states the combined result is more than 2× fewer floating-point operations per token than a standard transformer of the same configuration — which translates directly to less battery drain and less heat on a small device.

📊 Benchmarks & the DeepSeek V4 Flash Claim 10:03

The comparison was set up fairly — and the video is careful to say exactly what was tested. Tool-calling suites: Mobile Actions (961 rows, phone commands → Android intents), DroidCall (200 rows, some requiring two calls in the correct order), and BFCL v4 (3,641 rows, Berkeley function calling, including a check that the model doesn't call when no tool applies). Extraction suites: DSTC8, Snips Gold, Snips SevenWay.

The headline results, with their crucial caveat:

• Tool calling: beats models 10× its size on mobile tool-calling benchmarks.
• Extraction: matches models 2–3× its size.
• "Passes DeepSeek V4 Flash": fine-tuning on DroidCall lifts every subnetwork by 18–36 points, and from the 4-layer subnetwork up (starting at just 29M parameters), the fine-tuned model passes DeepSeek V4 Flash — specifically on DroidCall and Mobile Actions.

One more comparison detail the video flags as fair but worth naming: baselines ran at full 16-bit precision through a standard engine, DeepSeek V4 through its actual cloud API — while Needle's subnetworks ran through the real 2-bit quantized binary that actually ships. So Needle was tested exactly as it runs on a device, against models in their strongest form. Beating a much larger general-purpose model at a narrow task it was never optimized for is still genuinely useful — it's just a different claim than "beating a frontier model outright."

💻 Developer Workflow 12:52

Getting started is a standard Python pip install, after which the model is fetched once from Hugging Face and cached locally — nothing else to build or configure. A tool is declared by decorating a regular Python function: the signature tells Needle the argument types, and the docstring becomes the description the model reads. A single run() call handles the whole loop — decide which tool to call, execute your function, feed the result back, return a final response with results attached.

Two features stand out for reliability:

• Triggers: regexes matched against the user's text that, on a match, restrict decoding to the matched tool and guarantee a call even below the confidence threshold — so "turn the lights on" always reaches the right function.
• Structured extraction: declare the shape with a Pydantic model, call extract(), get back a fully typed object — no manual parsing.

Every response is one consistent JSON object: the function calls made, a written-out reasoning trail, a calibrated confidence score from a dedicated learned head, plus prefill/decode speed and peak memory. The confidence score has a built-in floor of 0.1, below which a call is held back into a "suppressed calls" field instead of executing. Cactus recommends a three-tier policy: above ~0.7, execute immediately; mid-range, show the user and confirm; nothing returned, treat as a genuine refusal. Fine-tuning uses LoRA on the frozen full 20-layer base, then exports a compact 4-bit file at whatever ladder depth you deploy to.

📦 Platforms & Business Model 15:52

Deployment coverage is unusually broad for a release this size: pre-built engines each under 1MB for macOS, five Linux architectures, Windows, three Android architectures, iOS, tvOS, watchOS, a browser WebAssembly build, and even a WASI component target for embedded environments.

The business model is open-core: Needle 3's weights sit on Hugging Face and the full source + inference engine on GitHub — free to download, run, and build with today. The paid layer is the Cactus platform, which sells access to curated datasets, the 2-bit quantization process that shrinks the shipped model, evaluation tooling, and full-depth fine-tuning infrastructure on Cactus's own training pipeline. The free model is a strong general-purpose starting point; the paid platform is where a company shapes it around its own products and tools without building a training pipeline from scratch.

⚖️ Verdict: Where It Makes Sense 18:12

Use it for anything that acts on a voice command or a short piece of text entirely offline: smart home apps, wearables, a robot following instructions without a network, an embedded product with real hardware constraints. The combination of tool calling + extraction + embeddings in one tiny binary, plus the ability to pick exactly how large a model you need, solves a specific problem general-purpose models were never designed for.

Don't use it for a general chat assistant that holds long conversations, answers open-ended questions, or writes creatively — Cactus has never claimed otherwise. It's a specialist in a narrow lane.

The single most practical piece of advice in the video: don't decide from the benchmark charts on the release page. A benchmark suite built by the company that made the model will always look favorable to it. Bring your own tool definitions, your own real schemas, and your actual use case, and test Needle 3 against that.

💡 Key Takeaways

1. One binary, three jobs. Tool calling, structured extraction, and embeddings in an 8–29MB file, entirely on-device — no cloud, no network, no per-call cost.
2. The intelligence ladder is the real novelty. One set of weights where every depth from 2 to 20 layers is a deployable model, from smartwatch to flagship — no per-size retraining.
3. The engram makes it cheap to run. Most parameters sit in an n-gram hash memory, so a 121M model does the arithmetic of a 50M one — ~2× fewer FLOPs per token.
4. Guaranteed-valid output. A byte-level decode grammar constrains generation, and a calibrated confidence head (floor 0.1) suppresses low-confidence calls.
5. "Passes DeepSeek V4 Flash" is a narrow claim. Fine-tuned on DroidCall, a 29M subnetwork beats it at that task — not a general win. Read the fine print.
6. Real traction, real hardware. 11.3k stars, Apache 2.0, and Pebble ships it in the screenless Index Ring where offline action is non-negotiable.
7. Open-core is the healthy sign. Free weights + a paid fine-tuning/quantization platform gives Cactus a reason to keep the project alive.
8. Specialist, not generalist. Excellent for offline act-on-voice; useless for open-ended chat. Test it on your own schemas before committing.

🔗 Resources & Links

• 🐙 github.com/cactus-compute/needle — source + inference engine
• 🤗 huggingface.co/Cactus-Compute/needle3 — model weights
• 📄 cactuscompute.com/needle — release notes & architecture diagram

Source video: youtube.com/watch?v=qiZITfI0wQE. Note: a companion piece to the earlier "Needle 3: You Don't Need an LLM for Function Calling" deep dive, which covered the hands-on demo and gotchas — this one covers the architecture, benchmarks, and business model.

⏱️ Timestamp Index