🎯 The Question: Speed or Calibration? 0:00
Jev, from TypeSafe (with $40M in funding), takes text and hands back structured answers — category, sentiment, intent — in 70–500ms. But what got people wasn't the speed; it was the confidence score next to each answer: "this is a billing complaint, and I'm 91% sure." Within hours of launch came the inevitable post — "you can use any LLM just like Jev" (260 upvotes) — plus Codacus's 306ms home-GPU demo and Harsha Gondala's two-hour rebuild that crossed a million views.
The question the whole video is built around: is the speed the hard part, or is it the confidence score? "I got some of this wrong before I understood it, so I'll point out where I tripped up."
⚙️ How Jev Mode Works: Parallel Constrained Decoding 1:54
Normal LLM classification generates tokens left-to-right: open brace, quote, "category", quote, colon, quote, "billing", quote, close brace — every token a separate pass through the weights. Jev mode inverts it: give the model the prompt and schema and say "the answer is one of these five." Instead of generating, the model evaluates each candidate option and returns the winner with a score — parallel constrained decoding. Read the prompt once and cache it; every candidate is a short branch off that cache; all branches evaluate in one batched GPU decode. "The model is shading a bubble on a multiple-choice test instead of writing an essay."
The fork is two commits by Codacus, using llama_memory_seq_cp to copy the cached context per branch, read logit probabilities at each token, and pick the highest-scoring path — with options sharing a starting token ("very negative" / "very positive") stored as a tree that splits where they diverge, "a genuinely elegant bit of engineering."
🎭 The Confidence Problem: Masked Logits 4:21
Here's the sharpest insight in the video: "the speed is the packaging; the product is the 0.9." That probability is what TypeSafe raised $40M to build — and it's hard for a reason. Parallel constrained decoding forces a choice even when the model is confused. The counterintuitive example: a model reading a support ticket assigns 30% to "billing," 10% to "technical," and 60% to the word "I" — because it wants to say "I think this is about billing, but…". Block everything outside the list, throw away that 60%, and renormalize: billing becomes 30/40 = 75%, technical 10/40 = 25%. "The model was mostly confused, and the output says 75% confident."
The open-source fork does exactly the naive thing — mask, renormalize, return a number — and one test item in the benchmark had all three fields wrong, "every one of them at 98% confidence or higher."
📊 Does the Confidence Actually Work? 6:47
InsiderLLM actually measured it rather than hand-waving. Out of 47 items, picking got 21 right, writing got 23 — both under half on a 27B three-field classifier, "which should make us humble about both." But the confidence score did carry signal: wrong answers had a lowest-field probability averaging ~0.62, right answers ~0.8. That gap is real enough to build an escalation pipeline — run the cheap local model in Jev mode, and anything below a threshold (say 0.8) escalates to a bigger model.
The math is where the demo meets reality. Gate at 0.8: you catch 21 of 26 wrong answers (great), but you also bounce 8 of 21 correct ones upstairs — that's 29 items, or 62% of traffic, going to the expensive model, and of the 18 the cheap model keeps, 13 are right (72%). "Not a disaster, but not the slam-dunk cost-saving story the demos suggest." Codacus recommends this exact pipeline in his video, but never tests the threshold — "it's just a number picked to look reasonable."
🏗️ Which Models Keep the Single Pass 9:30
"Any LLM" is technically true and practically misleading — the fork loads whatever llama.cpp loads, but they don't run the same:
| Architecture | Jev-mode behavior |
|---|---|
| Plain attention (e.g. 1.5B Qwen, 28 layers) | Best case — cache shared across every branch, 128 parallel sequences nearly free on an M4 Max. Works as advertised. |
| Sliding-window (e.g. Gemma 4 12B, 40 of 48 layers) | Window allocated per sequence — the fork's README says cap it at ~12 branches on a 12GB card. |
| Hybrid recurrent (e.g. a "dense" 27B with 48 linear-attention + 16 full-attention layers) | Loses the single batched decode (state per sequence), splits into several passes — a 4B hybrid was no faster than a 12B dense. |
| MoE with offloaded experts (e.g. a 35B A3B) | Copies expert weights across PCIe on any batch over ~30 tokens — never got under a second. |
The twist that "bit" the author: InsiderLLM called their 27B model "dense, no experts," so it seemed safe — but its config file shows 48 linear-attention and 16 full-attention layers, i.e. a hybrid. Their whole benchmark may have been hitting the slow path without anyone noticing. "The lesson: look at the model's config file before running any benchmarks, so we know what we're actually running."
⚠️ The Five Traps 11:37
- Cache invalidation from alternating schemas. The fork caches one prompt at a time; alternate between a sentiment and a category schema and every request is a cold start — 4.5× slower. Fix: group requests by schema.
- Branch memory allocation. Branches cost memory. On a 24GB card with 22GB committed to weights, 16 branches failed and it fell back to its minimum of 3 — plan for it on small cards.
- Fields can't see each other. Branches fork from the cached prompt, not from other fields' answers — an
is_adultfield can't read theagefield. A 30-field schema with dependent fields lost to plain JSON generation. - Cold-start latency. The first call is slow (~2s vs ~1.25s for writing) — warm the cache before benchmarking or the numbers mislead.
- The threshold nobody tests. Low-confidence answers near 0.5 flipped between server sessions. Before any gate ships, label a few hundred of your own examples, bucket them by confidence, and find where each bucket hits the accuracy you need — that's your threshold, and everything under it goes upstairs on purpose.
⚖️ The Verdict: Jev vs Small Models 13:46
Before reaching for the fork at all, check whether stock llama-server is enough: it already returns token probabilities via n_probs / logprobs, which is sufficient for a single yes/no field (a community wrapper reports ~23ms median on a Qwen 3 8B). The fork only adds what stock doesn't: batched decoding, multi-token labels, and probabilities normalized across the candidate set.
The humbling final point, from the video's closing: if your labels are stable and you have training data, a 22M-parameter classifier beat Jev on the same set by 13 points — in 8ms on a CPU. The real decision tree isn't "Jev vs the fork" — it's whether you need a calibrated decision model at all, or whether a tiny trained classifier does the job for essentially nothing. The fork is a legitimate speed hack; it is not a confidence hack, and treating it as one is the mistake the whole video exists to prevent.
💡 Key Takeaways
- Speed is the packaging; the 0.9 is the product. The fork copies the speed; it does not copy the calibration TypeSafe raised $40M to build.
- Most of the 11× is boilerplate. Picking ≈ one token of writing (298ms vs 275ms + 26ms/token); the "reasoning field" was the expensive part.
- Masked logits manufacture confidence. Forcing a choice and renormalizing over your options turns a confused model into a 75%-confident-looking one.
- Confidence carries some signal — barely enough to gate on. Wrong ≈ 0.62 vs right ≈ 0.8, but a 0.8 gate bounced 62% of traffic to the fallback.
- Even Jev is miscalibrated. 80% accuracy at 88% confidence on Banking77 — 8 points overconfident until temperature-scaled.
- "Any LLM" depends on architecture. Attention models share the cache free; sliding-window, hybrid, and offloaded-MoE models lose the single pass.
- Check the config file. InsiderLLM's "dense" 27B was secretly a hybrid — their benchmark may have hit the slow path.
- Calibrate your own threshold. Label a few hundred examples, bucket by confidence, and escalate below where accuracy drops.
- For stable labels, skip the LLM. A 22M classifier beat Jev by 13 points in 8ms on a CPU.
🔗 Resources & Links
- 🐙 github.com/thecodacus/llama.cpp (parallel-decision) — the fork under analysis
Source video: youtube.com/watch?v=Oi4JPj58lUA. A critical counterpoint to the earlier "Jev Mode in llama.cpp" deep dive — this one stress-tests the claims with InsiderLLM's RTX 3090 measurements.