llama.cpp's Expert Cache: The Real Story Behind the 4× MoE Speedup

0:00The setup: a 24 GB RX 7900 XTX, a 125B-parameter model, hours of tuning — and 9.7 tokens per second, slower than a person reads. Then on October 7 a developer's pull request landed in llama.cpp, and the same card/model/quantization hit 37.2 t/s — a 3.8× jump, rounded to "4×" by every thumbnail this week. The creator's verdict: this one deserves the hype. But the headline hides three or four things that can make your machine's result completely different.

What MoE actually is

2:47The model at the center is Qwen3.8-Flash-Next, a mixture-of-experts (MoE) model — 125B total parameters, but not all used at once. 48 layers × 512 experts per layer = 24,576 experts, and per token a router picks just 10 per layer — 480 experts, ~2% of the model. The other 98% sits idle. That sparsity is what makes MoE efficient on paper and obnoxious on consumer hardware at the same time.

Why it's slow

5:01When experts live in system RAM instead of GPU VRAM, every token forces the CPU to fetch the chosen expert's weights, run them, and ship the result back over PCIe — while the GPU idles. That's 48 memory-to-GPU handoffs per token, ~470 every second at 9.7 t/s. Half the generation time isn't compute at all; it's data shuffling. The GPU is being used as an expensive traffic light.

The old way

6:14llama.cpp already had an answer: the whole-layer offload flag (the video calls it n_cpu — "npu 48" = offloading 48 layers). It moves entire expert layers to the GPU: if a layer's 512 experts fit, they all go; if not, they all stay in RAM. Binary. A 24 GB card squeezes four or five full layers on and leaves the rest in memory. The waste: you pay VRAM rent for 512 experts per layer while only 10 run. "Buying a 512-slot parking garage to store 10 cars."

The expert cache

7:21Developer am17an (the captions garble it as "Lick 77" / "Lich 77" / "Le77") proposed — and llama.cpp merged Oct 7 as PR #29887 — moving individual experts instead of whole layers. Carve a free-VRAM pool (say 16 GB) as an expert cache; keep a living LRU ledger of recently used experts; if the router asks for an expert already in the pool, run it on-GPU; if not, copy just that one expert (~2.9 MB at 4-bit) over the bus and evict the least-recently-used. The key insight: the router isn't random — locality means certain clusters of experts get called constantly for a given task (code, narrative, arithmetic), like a restaurant keeping the pasta pot on the stove all night.

The honest numbers

9:41The 3.8× is real but measured against a very specific baseline: the same card with the model dumped entirely on CPU (n_cpu 48, ~zero experts in VRAM), then the cache filling 16 GB with ~a third of the experts. Going from "zero VRAM used for experts" to "16 GB used" naturally gives a big number. llama.cpp's own docs say 2.2×, measured against the default behavior that already crams whole layers onto the GPU — a fairer comparison, and still excellent. Then there's 1.55×: am17an's experimental branch vs. what actually merged into mainline, on the same hardware. Two reasons for that gap:

• Pinned memory. When copying a missing expert, pinned RAM (the OS promises not to move it) copies ~24 GB/s vs ~9 GB/s unpinned — one developer's decode went 49.4 → 58.1 t/s from that alone.
• The routing loop. In merged mainline, the router's decisions bounce back to the CPU at each of 48 layers — one timed run found 13–14 ms of a 17–19 ms token was pure CPU handoff. am17an's branch moves two critical ops entirely onto the GPU so it never asks the CPU.

The Oct 16 fix

13:25One day after the v0.7.0 release, an Nvidia engineer merged PR #30449 — Windows silently stops honoring large pinned-memory requests and falls back to ordinary memory, and this caps the max pinned buffer. On an RTX 5090 under Windows with multi-token prediction, that fix alone pushed 37.76 → 62.73 t/s. So part of the branch/mainline gap is already closed on Nvidia/Windows; the routing issue is still open, and nobody has rerun the full head-to-head since — the story is still moving.

Who wins, who loses

13:25Everything comes down to hit rate — the % of expert requests the GPU finds already in cache. It matters more than teraflops, RAM speed, or quantization. At 4-bit, Flash-Next has ~65 GB of expert weights; you need roughly a quarter of them (~16 GB of cache) to reach a useful hit rate, on top of attention and context. 24 GB cards can work; 32 GB is the sweet spot. One 5090 + 128 GB RAM owner went 27 → 50.8 t/s at 61k context; another went 23–24 → 40–45 t/s. The break-even is ~85% hit rate — below that, the miss overhead costs more than just computing on CPU. Small cards get hurt, not helped:

• 8 GB RTX 4060 laptop: 23.4 → 15 t/s (cache), → 13.4 at 2 GB; prompt reading 45 → 23 t/s.
• 12 GB RTX 3060: 32 → 12 t/s with an 8 GB cache.
• 12 GB RTX 4070 Super: no cache 14.8 t/s; 4 GB cache 12.9; 2 GB cache 11 — every size slower than none.

Half a gigabyte of cache on an 80B model was measured 24% slower than no cache. Rule of thumb: if you can't dedicate ~a quarter of the expert-weight size to the cache after attention + context take their cut, don't turn it on.

The hidden costs

16:43Two more things that don't show in a decode headline. First, your PCIe slot: every cache miss travels the bus, so x16 Gen4 (~31.5 GB/s) has headroom, but a secondary x4 chipset slot (~7.9 GB/s) caps you around 58 t/s at 90% hit rate, and Gen3 is worse — an RTX 2080 Ti owner found whole-layer placement faster than the cache (58.5 vs 53 t/s; 70.9 vs 46.1 with MTP). Check yours with nvidia-smi -q and look at "current link width/speed." Second, the cache only helps decode (small batches ≤32 tokens) — it does nothing for prompt processing, and the VRAM you gave the cache is VRAM prompt reading can't use. llama.cpp's own 2×4090 test: decode +1.85×, but prompt reading fell to 0.73×. 20:13And a coding agent that resends full project context every turn reads that context every time — 30% slower reading can outweigh 2× faster generation. Benchmark the workflow, not just decode. Speculative decoding is the third wrinkle: burst verification floods the cache with expert calls it wasn't sized for; llama.cpp's multi-token prediction made Flash-Next 27% slower when experts were in RAM (am17an's branch eked a 20% MTP gain only because its cache was big enough). Finally, one 5090 owner saw the model loop across four runs with the cache on — likely an unrelated sampling issue, but if your model loops, disable the cache first.

The 15-minute test

21:47Find your own number: (1) update to release b11480 or newer — the Oct 16 Windows pinned-memory fix matters; (2) run plain auto GPU assignment (or hand-tune whole-layer offload until VRAM is nearly full) and write down both decode and prompt speed; (3) add the cache flag --moe-cache-mib N with most (not all) free VRAM, leaving a few hundred MB for the OS — and run two or three warm-up prompts first, because a cold cache always looks worse; (4) set log level 4 and read the hit rate after — under 85% it's costing you, 85–90% is gray, 90%+ genuinely pays, 95%+ is real firepower; (5) compare against your best-tuned whole-layer split, not against zero — a big speedup from a blank baseline tells you how bad the starting point was, not how good the cache is.

Strata vs llama.cpp

23:56If your only goal is running Flash-Next fast and you have no deep reason to use llama.cpp, the dedicated runtime Strata (~31k★) is optimized for exactly this model family — and the numbers aren't close: a 12 GB RTX 5070 hit 62 t/s on Strata's 3-bit build vs. 37.2 on am17an's faster-than-mainline branch on a 24 GB Radeon; the one head-to-head measured 85.88 t/s (Strata) vs 58.1 (best llama.cpp), and prompt reading 2,320 vs 451 t/s. The catch is the "but": Strata runs one model family; llama.cpp runs basically everything — a new model tomorrow is supported within a week. Specialist vs. generalist, and neither is wrong.

Claims checked

Sources & further reading