Jev Mode in llama.cpp: Choose Instead of Write — Any LLM as a Millisecond Decision Model

🎯 The Claim: Milliseconds, No Hallucination 0:00

Jev answers in milliseconds, costs ~4¢ per million input tokens, and its makers say it can't hallucinate — but you need a waitlist spot to try it. The setup is deliciously asymmetric: TypeSafe spent ~2 years building it in stealth; within hours of launch, one developer open-sourced his own version built in 2 hours on an open model you can download today.

The questions that drive the whole video: what is Jev actually underneath? Is anything fundamentally different about it? Can you run the same thing in llama.cpp with models that already exist? And the big one — can the GPU you already own hit Jev's milliseconds mark, with the same type safety?

🔍 What Jev Actually Is 0:47

The contrast is drawn against how LLMs work today. An LLM "runs token by token, like a writer writing out the whole story, one word at a time." Ask it for structured JSON and it writes every brace, every field name, every value, one token at a time — taking seconds, and unless locked down it can miss a field, invent a field name, or wander off entirely. Jev answers everything in one pass: instead of writing the field names and values, it only produces the values; the system collects them, builds the JSON, and hands you one finished object in milliseconds.

It's pitched as "System 1" (from Kahneman's Thinking Fast and Slow): the fast reflex that answers before you've thought about it. It can't make up an answer because the system won't let it produce anything that isn't in your schema. The author went and read the published training method — RLCD (reinforcement learning for calibrated decisions) — and his read is pointed: it explains how the model gives honest confidence numbers, but "it doesn't explain the millisecond responses and it doesn't explain the no-hallucination claim." And the general idea isn't new — diffusion Gemma already fills masked spans in one shot. If that's all Jev was, "it's really not that impressive." What got interesting is what happened next.

🧠 The Secret Sauce: Choose Instead of Write 3:43

Hours after Jev's launch, Harsha Gondala open-sourced a repo with the perfect one-liner: "They were building in stealth for 2 years. I was building in stealth for 2 hours." What he put up wasn't a new model — it was an ordinary Qwen 2.5 1.5B, the same weights anyone can download. No retraining, no special architecture, just a different way of running it. It came back ~6× faster than normal decoding, and every answer matched the schema.

The mechanism is the heart of the whole video. Every time a model produces one token, it runs the entire network and scores every word in its vocabulary to decide what comes next — then picks one, appends it, and repeats. That's why writing is slow: a 30-field JSON answer is ~300 trips through the model, most of them spent producing quote marks, colons, braces, and field names you already knew. (Grammar-constrained decoding fixes broken JSON but not the speed — the model still writes field names one token at a time, and can still give you valid JSON with the wrong answer inside.)

The trick: don't ask the model to write the JSON at all. Ask one question — "what's the value for this field?" — and have it answer in a single token. For a field whose allowed values are technical/billing/shipping, map them to A/B/C so the answer is one token; read only the scores of those option tokens and ignore everything else, which gives you a normalized confidence per option. (You don't even need the letters — options usually differ at their first token, so you read the real words' scores; only when two options start identically, like "very negative" vs "very positive," do you read one more token.)

⚙️ Jev Mode in llama.cpp: /v1/decision 9:06

So the plan was to put that idea where "the rest of us already live" — llama.cpp, which runs on Nvidia, AMD, CPU, Mac, and basically every model you've already downloaded. The addition is a new /v1/decision endpoint on llama-server: send instructions, a schema, and the current state; get back every field with a value and a confidence score. Same server you chat with, same model in memory, one extra way to ask it things.

Two implementation details matter, and both are worth stealing for your own engine:

• The "where do you cut the line" bug. The first version's answers were subtly wrong. The questions live in the instructions, but what you score is the JSON — the model gets the opening brace, field name, and colon, and you read what it puts next. Cut after "colon + space" and score "true", and you're asking about something the model would never have written (it writes colon+space+word as one piece), so confidences come back skewed. The fix: tokenize the whole line (field name + value together) and cut where the options actually start to differ.
• The KV-cache trick that makes it parallel. The instructions, field list, and allowed values never change — the context does. So you read the instructions and schema once and keep that KV cache; every later request starts from a model that has already read them, adds only its own few tokens (field name + colon), and reads its answer. All questions point at the same shared cache, so the expensive pass is paid once and each extra question costs a handful of tokens. Then you can hand over eight requests in a single call — all through the model together, every answer back in the same pass.

The numbers: eight requests in one call landed ~12× faster than asking the same model to write the eight answers out as JSON — and roughly double the throughput versus sending them one at a time (about quadruple once you count the eliminated round trips).

🖥️ Run It Yourself 13:15

Setup is ~10 minutes, all open. Clone the fork and switch to the parallel-decision branch — it is not on main, and forgetting this step builds a perfectly ordinary llama.cpp. Build for your hardware (CUDA / Metal / ROCm / CPU). Grab any GGUF you like — the video uses Gemma 4 12B for judgment calls, since it fits a 12GB card. The one setting that matters lives in a presets file: how much room to keep for parallel questions ("give it a dozen, and a dozen questions can be in the air at once").

The decision playground is a browser app with no backend of its own — it talks straight to your llama-server, so your data goes nowhere. Point it at your server, pick a preset (the support-ticket one shows schema + allowed values on the left, the customer's message below), and fire the same question two ways: the decision endpoint versus the same model writing JSON by hand. 300ms versus 3.5 seconds, the same 12 answers — and every field comes back with a confidence number, which is the part that matters more than the speed: some land at 100%, some sit lower, and that's the model telling you where it was guessing.

🔬 The Model Hunt: What Loves It, What Breaks It 17:03

Throwing other models at it surfaced the real engineering, and it's honest about the failures:

🧭 What You'd Actually Use It For 20:59

The architectural frame is Kahneman again, but applied literally. Walking down the street reading a message: your legs, balance, the oncoming people, the curb — that's System 1, running a dozen little routines at once and only interrupting when it hits something it can't handle; reading and composing the reply is System 2, one thing at a time. A car turns into your path and it's System 1 that notices first, because System 2 has been glued to the screen.

The practical mapping is a two-tier pipeline:

• System 1 (Jev mode): a support inbox of thousands of emails — category, priority, human-needed yes/no — processed in bulk, milliseconds each, with a confidence number attached. Confident answers go straight through.
• System 2 (normal LLM): anything the model wasn't sure about, handed to a bigger model that thinks slowly, one at a time — now a handful of emails, not thousands.

The same shape works for game agents (next move several times a second), home automation (every light and blind is a field in one schema), and moderation/routing. Model choice: a 2B on an 8GB card for sorting and routing; a 12B (Gemma 4) when being wrong costs something; and switch back to the slow-thinker mode for the ones it flags as unsure.

⚖️ What It Comes Down To 25:01

What's underneath Jev? Nobody outside TypeSafe can say. The founder has claimed he co-invented the reinforcement learning behind ChatGPT and later called chat "a detour" — connect the dots yourself. But what matters is the demonstrated fact: an ordinary open model, asked a different way, does the same job — in llama.cpp, on files already on your drive, on the card you already own, in milliseconds.

💡 Key Takeaways

1. The trick is "choose, don't write." Instead of writing field names + values token-by-token, score only the allowed option tokens — one token per field, one pass for all of them.
2. It's not a new model. Harsha's version was an ordinary Qwen 2.5 1.5B, no retraining — just a different way of running it (~6× faster).
3. Prefill beats generation. Parallel decisions ride the prefill path over a shared KV cache: instructions read once, each question adds only a few tokens.
4. Where you cut the line matters. Score the whole "field name + colon + value" as one tokenized piece, or confidences skew.
5. 17.3ms/decision in bulk on a 2GB model in 2.4GB VRAM; 306ms vs 3,415ms for the same 12-field JSON.
6. Not every model gets fast. Attention-only models fly; hybrids and MoE models hit batching walls (recurrent same-length constraint, expert weights over the bus).
7. Wrong ≠ hallucination. Type safety guarantees format, not correctness — verify the model knows your task.
8. Fields are independent. Parallel questions can't see each other, so interdependent rules need a plain JSON-writing model.
9. The right shape is two-tier. Jev mode for the confident bulk, a slow thinker for the unsure remainder.

🔗 Resources & Links

• 🤗 huggingface.co/harshatheg/Qwen-2.5-1B-RLCD — Harsha's model, the original proof (Apache 2.0)
• 🐙 github.com/thecodacus/llama.cpp (parallel-decision branch) — the llama.cpp fork with Jev mode
• 🐙 github.com/thecodacus/decision-playground — the browser playground

Source video: youtube.com/watch?v=bcGO7xre46o. The most technical of the Jev-series deep dives — this one covers the inference-engine implementation.

⏱️ Timestamp Index