🎯 The Idea: A Router That Decides for You 0:00
The build: a model router running as a local endpoint. You send it a request; it interprets what you asked — via Jev or an open Jev clone — decides what category the request falls into and which model it should go to, and also detects PII so sensitive input can be forced to stay local rather than shipped to the cloud.
The model lineup spans both worlds. Locally: MiniCPM 5 2B (a tiny 2B model served via SGLang — or LM Studio, or Ollama) and Qwen Image 2.1 (just released the day before recording, image gen + editing; currently non-commercial license). In the cloud: DeepSeek V4.1 Flash on OpenRouter, plus optional web-enabled and "hard" tiers (Opus 5). The router's job is to pick among them per request.
📝 Jev Recap: An If-Statement That Understands Language 1:45
Jev is TypeSafe's system-one model — it "doesn't generate text at all." You give it a state (whatever you want judged) and a set of typed questions, and you get back typed answers with probabilities. The point is that your code can then make plain conditional decisions on it: "if the choice comes back this, do this; if the probability is above this percent, do that." No worrying about a model emitting valid JSON or complying with a prompt.
🗂️ Three Question Types → Three Router Jobs 3:47
Jev (and most open clones) offer three question types, and they map onto a router almost perfectly:
| Question type | What it returns | Router job |
|---|---|---|
| Choice | Picks one option, probability per option + confidence | Lane: chitchat / simple question / rewrite / code / reasoning / image |
| Score | Position on an ordered scale (up to 10 levels) | Difficulty: can the 2B model handle it, or escalate to a bigger tier? |
| Null | Yes/no (true/false) + probability of yes | PII gate: does this contain private data? Force local if so. |
Three more design points matter. First, all three questions go in one request against the same state and Jev evaluates them in parallel — three questions costs about the same as one. Second, the confidence value lets the router know when it's unsure, so instead of guessing it falls back to a safe default. Third, there's a leak to be honest about: if you send a prompt to TypeSafe to ask "is this private?", you've already leaked it. The fix comes later — swapping in an open clone that runs fully locally.
🎬 The Demo: Lanes, PII, and 331ms 5:59
The UI offers manual model selection or Auto, where Jev monitors and decides. The live results tell the story:
- "Hey, how's it going?" → difficulty 0, no PII, routed to the local 2B model — decided in 331ms.
- "Write Python to deduplicate stuff" → difficulty up, routed to DeepSeek (code lane is configured to go cloud).
- A fake API key → PII detected, forced local — and it's not doing anything with the key, just demonstrating the gate.
- A person's name or phone number → PII caught at 0.96 confidence.
Peeking at what's actually sent to Jev reveals the exact prompts: the choice question carries the six lane classes, the score question carries levels from trivial (a 2B handles it) up to frontier (needs the best model), and the null carries the PII instruction (personal, confidential, financial, medical, credentials, API keys, passwords). A second null decides whether web access is needed.
🏗️ Architecture: FastAPI, Lanes, and Conditional Logic 11:47
The stack is deliberately boring and small: a Next.js UI talks to a local FastAPI server. One call does the deciding and the routing, streams back the winner's response, and falls back if needed. Everything is logged to SQLite (Jev decisions, messages, even generated images), with health checks and OpenAI-compatible endpoints throughout.
Routing is expressed as preconfigured lanes with an order of preference — for "hard," if Opus is on, it routes to Opus first before falling back to DeepSeek. The conditional logic is plain: privacy flag, difficulty, token count, web-needed. Two nice touches: a tools registry (Qwen image is registered as a tool, and the judge itself can be swapped between the web Jev and an open clone), and prompt rewriting — the image prompt "a sandy-colored macaron" gets rewritten by MiniCPM into a richer prompt ("a sunlit meadow…") before hitting the image model.
🔓 Swapping to SemIf: The Open, Local Jev 15:06
Now the judge is swapped from the cloud Jev to SemIf — an open, local Jev-style model (openjev.com). The privacy test runs again and SemIf nails it, fully local, in 88ms. The input/output format differs slightly from TypeSafe's, but it does "the exact same kind of thing" — difficulty, private flag, lanes. A web-search request routes through OpenRouter's web model with DuckDuckGo under the hood.
📊 Stats, Costs & Verdict 16:46
Since restarting, the router handled 53 requests and answered 75% of them locally — which is where the savings come from. It did spend money (on Jev calls and DeepSeek), but the dashboard shows the savings of routing the majority locally, along with response times and how many prompts were private. The setup also supports full multi-turn chat, and could be configured to stay on a cached Opus session for long conversations.
The closing takeaway is the substance: whether you use cloud Jev or an open local clone, this rethinks how you structure agentic systems — decisions happen fast, cost a tiny amount, and scale. The entire day's demos cost less than one cent.
💡 Key Takeaways
- A decision model is a natural router. Typed answers + probabilities map straight onto if/else logic — no JSON parsing, no prompt wrangling.
- Three questions, one parallel request. Choice (lane) + score (difficulty) + null (PII) run against the same state in one call, ~331ms.
- Confidence enables safe defaults. Low confidence → fall back, don't guess; thresholds (privacy > 0.5, difficulty > 6) are plain code.
- PII gating works. Fake API keys and phone numbers caught at 0.96 confidence and forced to stay local.
- But don't leak to the judge. Asking the cloud "is this private?" already leaks it — swap in an open clone (SemIf) and run fully local (88ms).
- 75% of traffic went local. That's the real cost lever — route cheap, escalate only when the difficulty score says so.
- Decision-layer economics are absurd. Output is free, input ~4¢/MTok, and a whole day of demos cost under a cent.
- It's a rethink, not a widget. Jev (or OpenJev) changes how you structure routing, gates, and escalation across agentic systems.
🔗 Resources & Links
- 🐙 github.com/samwit/llm-tutorials — Sam Witteveen's tutorials (router code lands here)
- 🌐 openjev.com — SemIf, the local open Jev-style decision model used in the swap
- 🔗 typesafe.ai — Jev, the closed-source reference model
Source video: youtube.com/watch?v=ZR7anrL50xs. A companion piece to the earlier Jev deep dives (the hands-on test and the three-prototype tutorial) — this one is a full practical build.