🎯 The 144 TPS Mac Mystery 0:00
The setup is a familiar pain: a local model on a Mac, "I'll just go check Twitter real quick" speed. Then a tweet appears — a screen recording of Qwen3.8-27B running at 144 tokens per second on a MacBook Pro, caption "Same hardware. Nothing changed." The natural reactions are wizardry, lying, or "I've been doing it wrong for months." It turns out to be none of the three — and most of what's actually going on is actionable today on the hardware people already own.
The tweet came from Inco AI, showing their inference engine Splash. Within 24 hours LM Studio shipped Splash as a selectable backend, and a write-up went viral with the headline "48 GB MacBook Qwen at 74 tokens per second." The hardware genuinely didn't change — but the number underneath is doing more work than the number itself.
⚙️ What Makes Splash Fast: Two Models, Fused Kernels 1:13
Splash is an open-source (Apache-2.0) inference engine built specifically for Apple Silicon — and it's genuinely good engineering. The core bet is specialization: it runs exactly two models, Qwen3.8 27B and 35B, and nothing else. The philosophy is the inverse of Ollama/llama.cpp/MLX, which must cope with any model someone converted at 2 a.m.
- Fused kernels compiled to fixed shapes. Two models means every kernel is compiled to exactly those dimensions ahead of time.
- Fixed-layout binaries. Weights load straight from disk in a pre-baked layout.
- A memory planner that refuses to boot rather than die mid-generation — if a Mac can't run the model, Splash prints a memory report and exits, instead of crashing 9 minutes into a 400-line rewrite.
🚦 The Real Bottleneck: Memory Bandwidth 3:27
The reframe that unlocks the whole benchmark: an M-series Mac can do trillions of ops/sec — the marketing isn't lying — but a 27B model still feels like it types with oven mitts. The chip isn't straining. It's waiting.
Every token produced means dragging the entire model across the memory bus and into the compute units, doing the arithmetic for one token, then throwing the weights away and dragging all of them across again for the next token. A 500-word answer repeats that hundreds of times. The arithmetic was never the slow part — memory bandwidth always was.
And the numbers check perfectly:
| Chip | Bandwidth | Model (4-bit) | Decode ceiling | Splash claim | Multiplier |
|---|---|---|---|---|---|
| M5 Pro | 307 GB/s | 15.1 GB | ~20 tps | 74 tps | 3.6× |
| M5 Max | 614 GB/s | 15.1 GB | ~41 tps | 144 tps | 3.5× |
🎲 Speculative Decoding: Draft + Verify 4:47
"The way we make the model faster is by giving it more to do." The technique is speculative decoding: a tiny draft model (DFlash 2, weighing 1.2 GB) guesses the next few tokens fast, and the big 27B model — the one that actually knows things — verifies all of those guesses in a single pass instead of generating them one at a time. Guesses that survive are kept; the rest are thrown away. Either way, one trip across memory yields several tokens.
Run the numbers again with that in mind: Inco's draft proposes ~7 tokens per step, and roughly 3–4 survive verification on average. That's where 74 comes from on the M5 Pro, and 144 on the M5 Max. It's a 1.2 GB file doing exactly what draft models do — not the engine.
🔍 The Truth Behind "2× Faster" 6:50
The "2×" is measured against oMLX — Splash gets 74 tps, oMLX gets 38. That's the 2×. But Inco's own methodology note is explicit, and worth reading closely:
- Each engine ran at its own recommended settings, so the benchmarks show the combined effect of specializing the engine for its model — "the benchmark does not isolate the engine."
- oMLX had an open feature request for Qwen draft-model support at benchmark time — it supports draft models for Gemma, but not Qwen.
So the comparison is a setup with a draft model vs. a setup without one — speculative decoding against pure autoregressive decoding. It isn't Splash vs. oMLX; it's speculative vs. not. Inco documented all of this honestly; everyone downstream just stopped reading after the headline.
Community benchmarks bear out what "free" gets you: a 3090 went 31 → 41 tps (~30%), a laptop 5090 went 37 → 51, and a llama.cpp Mac user saw ~23%. None are 2×, but they're free, real, and available tonight. The reason Inco's number is 2×: they benchmarked SpeedBench coding tasks with reasoning mode on — about the most draft-friendly workload that exists (code is repetitive and predictable) — against a setup with no draft at all.
⏱️ 96 Seconds of Waiting vs 19 of Output 9:00
The number nobody quotes, published by Inco on the same page: on the 48 GB M5 Pro benchmark machine, a 32,000-token prompt with a cold cache takes 96 seconds to return the first token. Not 96 milliseconds — 96 seconds, watching nothing happen for a minute and a half before the model says a single word. (oMLX took 317 seconds on the same prompt, so Splash genuinely wins — but you're still waiting 96 seconds.)
Do the math that belongs on every benchmark table and never is:
| Metric | Cold cache | Warm cache |
|---|---|---|
| Time to first token (32K prompt) | 96 s | 282 ms |
| Writing time (54 tps × 1024-token cap) | 19 s | 19 s |
| Effective tokens/sec (send → finish) | 8.9 | 53 |
96 seconds of waiting against 19 seconds of output — and the 19 seconds is the part everyone was sold. A developer benchmark on an M1 Max showed the same effect: the interface reported 57 tps, but timed end-to-end the response ran at ~3 effective tps, because prefill was consuming 94% of the total time. The decode rate on screen has never once predicted how long you actually wait.
🔥 The Cache Changes Everything — Who Should Install 10:45
Replay that same 32K prompt warm, and the time to first token drops from 96 seconds to 282 ms. The effective rate jumps to 53 tps from 8.9. The cache is worth a 6× improvement; the draft model is worth 2× on a number that never captured the real experience. Inco led with the draft model when the cache was the better story.
And Splash's cache advantages are real. On long agent sessions — working through a codebase across many turns — the time to first token that you feel every single turn is 282 ms on Splash vs. 249 ms on oMLX... and Splash handled 16 concurrent 32K-token requests on a 48 GB machine, where general-purpose memory policies managed nine. If you run multiple sub-agents in parallel (increasingly how serious coding workflows operate), that throughput can't be had elsewhere.
Who should install Splash this week: you have a 48 GB M5, you run coding agents most of the day, and your sessions are long — for that, the cache wins are real and you'll feel every one, and it speaks OpenAI/Anthropic APIs so Claude Code, Codex, and OpenCode point at it with one config line.
Who should not bother: the requirements gate out almost everyone — M3 or newer, macOS 26.4, and 36 GB unified memory minimum (48 recommended), when most Macs ship with 16 or 24. The model format is proprietary (a 17.4 GB package with 4-bit weights, a vision encoder, and the DFlash 2 draft that only loads in Splash — your existing MLX/GGUF files just sit there). For everyone else: turn on multi-token prediction on the model you already have, set draft depth to 2 or 3, and time a real prompt cold and then warm — that's your actual benchmark.
💡 Key Takeaways
- The speedup isn't the engine — it's the draft model. A 1.2 GB DFlash 2 doing speculative decoding, not the fused kernels.
- Memory bandwidth is the ceiling. 307 GB/s ÷ 15.1 GB ≈ 20 tps max on M5 Pro; 74 tps is 3.6× that, which no kernel can explain.
- "2×" is speculative vs. autoregressive, not Splash vs. oMLX. oMLX had no Qwen draft model at benchmark time — Inco's own note says the benchmark doesn't isolate the engine.
- Inco documented everything honestly. The internet quoted the ratio and dropped the footnote.
- You already have most of it for free. Qwen3.8-27B ships a multi-token prediction head in the weights you downloaded.
- The benchmark workload is cherry-picked. SpeedBench coding + reasoning mode is the most draft-friendly workload that exists.
- 96 seconds to first token on a cold 32K prompt. 8.9 effective tps vs. the 74 tps headline — prefill dominates the wall-clock.
- Cache reuse is the real 6× win. Warm, the same prompt hits 53 effective tps — and Inco under-sold it.
- Stop reading the on-screen token counter. Run one prompt cold, one warm, time both with a stopwatch — that's your number.
- Splash is for a narrow slice: M3+, macOS 26.4, 36–48 GB, long parallel agent sessions. Everyone else turns on multi-token prediction for free.
🔗 Resources & Links
- 🐙 github.com/incoai/splash — the inference engine (Apache-2.0, C++, ~847★)
- 🤗 incoai/Qwen3.8-27B-Splash — the Splash model package
- 🤗 Qwen/Qwen3.8-27B — the base model with the built-in prediction head
- 🐙 jundot/omlx#1779 — oMLX draft-model support request (the other side of the 2×)
- 📄 Inco AI Splash launch post — the methodology note quoted throughout
Source video: youtube.com/watch?v=qZTmBhKz7b0. All figures are Inco AI's own published numbers; the memory-bandwidth ceilings are derived from Apple's M5 Pro/Max specs.