NCM Is Not the End of LLMs — Local Tests

0:00A Stanford model called NCM (Next-Concept Modeling) translates a German sentence into gibberish on its first pass — then snaps into clean English eleven passes later. The paper claims up to 31× faster generation and a tiny NCM model solving 86% of hard Sudoku puzzles (vs ~1% for an equally-sized autoregressive model), which had people calling it "the end of LLMs." Prompt Engineering read the paper, found two claims that were never actually tested, and ran the released models on a DGX Spark to check them head-to-head.

The memory bottleneck

1:50The problem NCM targets. Today's chatbots are autoregressive: they write one token, append it, then predict the next — a 1,000-token answer is 1,000 sequential steps. The expensive part isn't the math, it's the memory: on every step the GPU reads the entire set of weights out of memory to produce one more token. "The speed you see is mostly decided by how fast your memory is, not how fast your GPU does math" — the reason a local model writes slower than you can read. The obvious fix is to write many words at once, which is what diffusion models do (Mercury from Inception, Google's Diffusion Gemma): start with a page of blanks, fill a rough guess for the whole answer, then redraw the page again and again until clean. The catch: diffusion touches every position on every pass, and good quality needs 64+ passes. NCM asks — can you write in parallel without paying for all those full-page passes?

How NCM works

3:55NCM borrows the coarse-to-fine idea from image generation: draw a tiny blurry whole first, then sharpen. Two pieces. The compressor (a multiscale residual VQ-VAE) stores a passage as a stack of levels of discrete codes — the top level has a handful of codes holding a rough outline, each level below stores the missing detail, and the bottom level has one code per word. There are ~4,000 learned codes, and the paper calls each a "concept" — not a human idea, just a learned compression code. The generator is a plain transformer, but instead of predicting the next word, it predicts every code in a level at once, then moves down a level. A 9-level model takes 9 passes whether the answer is short or long (the German translation took 11 passes because that model has 11 levels). The key difference from diffusion: NCM's top levels are tiny, so early passes are nearly free, and each level only adds the detail it owns.

The two untested claims

7:19The paper's headline numbers were against diffusion models, and the coverage ran with them — but the video spots two gaps. First, every speed number is measured against diffusion models, never against a normal autoregressive LLM (the thing people actually run locally). Second, on text tasks like translation and open-ended writing, the plain word-by-word model in the paper's own tables still wins on quality. So the two questions that actually matter for local users were wide open: is NCM faster than a same-size LM on the same hardware, and where does the Sudoku win really come from?

Test 1: no streaming

8:38First, the paper's numbers reproduce — he gets "almost the exact same 86%" on hard Sudoku. But watching the board evolve pass-by-pass reveals the catch: NCM does not fill squares one at a time. It redraws the whole board every pass, even gets the starting clues wrong early, then snaps the entire grid into place at the end. The consequence for normal use: you can't stream an NCM answer the way you stream autoregressive text — nothing is readable until the very last pass finishes.

Test 2: speed flips

9:19The head-to-head with a same-size, same-shape autoregressive model: on one request at a time, NCM is ~4× faster on translation and ~2× faster on long-article summaries. But at 64 concurrent requests — what a real server does — the results flip: the autoregressive model beats NCM. The reason goes back to memory vs. math. On a single request, the GPU mostly waits on memory, so 9–11 passes beats 1,000 steps; on a batch, the weights load once and the GPU does math all day — and NCM does more math, processing ~4× more positions than words it writes, plus the released code re-runs the whole prompt on every pass (which is why long articles hurt it most). Bottom line from his small-model tests: a real speedup for one person on one GPU, a slowdown on a busy server — and that matters for agentic workloads, which run many agents simultaneously.

Test 3: the Sudoku mystery

11:19He trained his own diffusion model of the exact same size on the same puzzles and tested it on the same hard boards. Random fill order: ~25% (in line with the paper). But with one test-time change — fill the most-confident cell first — it got close to NCM, and with more passes it climbed to 95%. His read: the big Sudoku gap in the paper comes mostly from the order cells are filled, not the architecture. What NCM still has going for it: it gets coarse-to-fine "for free," where the diffusion model needs that extra decoding rule — so the likely path is combining the two.

Limitations

12:43The honest caveats. Every NCM model writes into a fixed-size box, so even a one-line answer runs every level and pays the full cost. The compressor sometimes swaps names and numbers when it encodes and decodes. The biggest model trained is under 1B parameters — no one knows if this holds at 27B or hundreds of billions. And the released code is evaluation-only: no training code, and the prompt-cache option is broken. Bottom line: a genuinely neat step forward for single-user speed, with real limits — "sketching the whole answer before writing the exact words" is the advance, not the end of autoregressive LLMs.

Claims checked

Sources & further reading