Deep Dives

Deeply technical walkthroughs for readers who already know the basics.

Page 1 of 1 · showing 1–17 of 17

The Question That Inspired LoRA: Intrinsic Dimension and Why Tiny Ranks Work

Julia Turc explains why LoRA works by going back to the question that inspired it: intrinsic dimension. Full fine-tuning of a 2.8T model needs 30TB+ of GPU memory, so LoRA constrains each weight update to a low-rank product B×A, freezing the pretrained weights. The 2018 Uber paper measured task difficulty via intrinsic dimension (pendulum 4, MNIST 750, humanoid 700), and Facebook's 2020 paper found RoBERTa fine-tunes with intrinsic dimension 322. Her demo fine-tunes Qwen3.8-27B as a prompt-complexity router: r=1 matches r=16 (both ~76%), and stable-rank analysis shows the median effective rank is just 3.

deep-dives

Julia Turc

Published2026-10-26

A Tiny Local AI Does REA: Qwen 27B + Hermes on One GPU

NetworkCoder sets up a fully-local reverse-engineering agent on a single RTX 3090: Qwen3.8-27B quantized, llama.cpp with 128K context and 8-bit KV cache, the Hermes agent driving REA (207 tools, tool search) and Ghidra. It cracks a stripped binary's save file (5/5) and writes a working license keygen (5/5), self-correcting, at ~40 tokens/sec with zero cloud bill. Covers the hardware budget, quantization, context limits, and the ethics of only testing your own binaries.

deep-dives

NetworkCoder

Published2026-10-24

llama.cpp's Expert Cache: The Real Story Behind the 4× MoE Speedup

llama.cpp merged a GPU expert cache for Mixture-of-Experts models (PR #29887), taking Qwen3.8-Flash-Next from 9.7 to 37.2 tokens/sec on a 24GB card. The honest comparison is 2.2× versus optimized whole-layer offload, and the break-even is an 85% hit rate needing about 16GB of expert cache — so small cards and slow PCIe slots get slower while big-VRAM cards win. Also covers the Windows pinned-memory fix (PR #30449) and the Strata runtime.

deep-dives

Kai

Published2026-10-24

JPEG XL: The Next Image Format, Explained

ThioJoe breaks down JPEG XL (.jxl) — the image format that just landed in Chrome 155 by default (joined by Firefox 158, and Safari since 2023). Walks the feature set plain JPEG lacks (alpha, >8-bit color, lossless mode, animation, HDR, regions of interest, tiling), the compression edge over JPEG/WebP/PNG, lossless JPEG↔JXL recompression for archiving, the AVIF rivalry, and whether it actually goes mainstream. Verified: Chrome 155's jxl-rs Rust decoder and Oct 6-7 stable ship, Firefox 158, Safari 17, the Windows JPEG XL Image Extension, and the 30-50% compression claim.

deep-dives

ThioJoe

Published2026-10-15

The 5 Levels of Text Splitting for Retrieval

Greg Kamradt's 69-minute foundational tutorial on chunking for RAG, walking the five levels of text splitting: (1) character splitting — fixed static lengths, simple but rigid; (2) recursive character splitting — a list of separators (paragraphs→newlines→spaces→characters), his recommended default; (3) document-specific splitting — markdown headers, Python/JS code, and PDFs where tables become HTML and images get text summaries via a vision model; (4) semantic splitting — embedding-distance break points (95th-percentile outliers) to split on meaning instead of position; and (5) agentic splitting — an 'agent-like system' that decomposes text into standalone propositions and decides per-proposition whether it joins an existing chunk (UUID + summary + title) or starts a new one. A bonus section covers alternative representations: multi-vector indexing (embed summaries or hypothetical questions), parent-document retrieval, and graph extraction via Diffbot. The through-line is the 'chunking commandment': don't chunk for chunking's sake — get the data into a form that retrieves value for the task. Frameworks verified (LangChain, LlamaIndex, Unstructured, Chroma, RAGAS); the video's specific model choices (GPT-4 Vision Preview, OpenAI embeddings) are dated as of 2026.

deep-dives

Greg Kamradt

Published2026-10-13

REA Hands-On: Cloning a Voice-to-Text App in Minutes (Step by Step)

Eric Tech's hands-on tutorial for REA (Reverse Engineer Anything, morluto/rea): the mental model (REA as a skill orchestrating Hopper/browser/mobile/hardware MCPs), the exact setup (Node v24+, npx rea-agents setup, install Hopper, approve the ~5 changes), and a real demo — he points Claude Code at WhisperFlow, a voice-to-text app he uses daily, and gets back a working clone with conversation history, a note-taker, a global shortcut, multilingual input, and fully local transcription. His thesis: 'code is no longer the moat — the brand, users, and data are,' citing PhotoCraft's Rust rewrite of the Adobe suite. The video's 'over 60,000 stars' is stale: the repo is now at 77,833 stars and 16,688 forks (GitHub API, Oct 11) — it tripled in two days since our first REA deep-dive (28.5K on Oct 9). His paid 'school community' is a self-promotional segment, flagged.

deep-dives

Eric Tech

Published2026-10-11

REA: The Fastest-Rising Repo Is a Little Scary (And a Real Test)

Rob Shocks (Switch Dimension) puts REA ("Reverse Engineer Anything", morluto/rea — MIT, 28.5K stars and climbing) to a real test: he builds his own "vault" app with a hidden unlock code, compiles it with every function name stripped, hands it to an agent running REA + Ghidra, and watches it find the check-unlock function among 242+ stripped functions, decompile it to C-like pseudo-code, and write a code generator that produces a working code (AGNT-1001). The nuance he stresses: REA doesn't recover source code (it's gone after compilation) — it returns pseudo-code, assembly, call paths, and network activity with evidence (source, confidence, unknowns), then the agent understands the logic and writes something new. The scary thesis: reverse engineering's friction was its protection, and REA deletes it — so "the code was never really the moat"; your moat is data, users, cloud security, ship speed, taste, and server-side logic. He pushes back on planes-falling doom and gives three warnings: legal (interop RE is generally protected but ToS often ban it, DRM is separate), it's not a sandbox (runtime capture runs with your permissions), and the repo is moving fast. Testbrite (sponsor) and Switch Dimension (self-promo) flagged. This complements the earlier Signal Coders deep-dive of the same repo with a hands-on, security-flavored angle.

deep-dives

Rob Shocks

Published2026-10-10

J-Space: Anthropic's Jacobian Lens for Reading LLM Thought

Arivu breaks down Anthropic's global-workspace interpretability paper — 'Verbalizable Representations Form a Global Workspace in Language Models' (arXiv 2607.15495) — and the Jacobian lens, or J-Lens, behind it. The starting point is cognitive science's global workspace theory: a theater where specialist modules in the audience post one item at a time to a single spotlight stage, which broadcasts back to everyone. Anthropic finds LLMs have an analogous 'J-space' — a shared vector space of activations — and the J-Lens reads it by measuring how the model's output reacts to nudges along each direction. Most directions are flat (the output shrugs); a handful are steep, and those are the concepts the model is actively reasoning with. The lens attaches human labels to the steep directions and lets researchers steer behavior by pushing along them. The caveats are real: superposition smears concepts across directions, and the sensitivity map is local — a snapshot valid only where you took it.

deep-dives

Arivu

Published2026-09-30

Splash's 2× Is Real — It's Just Not the Engine: The Memory-Bandwidth Math Behind Qwen3.8-27B on Mac

Inco AI showed Qwen3.8-27B at 144 tokens/sec on an M5 Max — and 74 on the M5 Pro they actually benchmarked. Both numbers are honest, and the gap between them is the story. Running the arithmetic on the vendor's own figures: an M5 Pro moves 307 GB/s and the 4-bit model is 15.1 GB, capping decode near 20 tps — yet Splash reports 74, 3.6× over the memory bus. No kernel beats the bus, so the speedup is the bundled 1.2 GB DFlash 2 draft model doing speculative decoding. The '2×' is measured against oMLX, which had no Qwen draft model at benchmark time — speculative versus autoregressive, as Inco's own methodology note admits. And the number nobody quotes: 96 seconds to first token on a cold 32K prompt (8.9 effective tps) versus 53 warm — making cache reuse the 6× win Inco under-sold.

deep-dives

Kai Explains

Published2026-09-26

Running Any LLM in Jev Mode: The Speed Is Real, the Calibration Isn't

Two commits to a llama.cpp fork can make an open model answer twelve questions in 306ms — inside Jev's own 70–500ms band. But Kai's critical breakdown shows what actually copies and what doesn't. The speed is real but mostly trivial: 'picking' costs about one token of 'writing,' so most of the headline 11× speedup is just deleting braces, quotes, and field names. The part TypeSafe raised $40M for — the calibrated confidence score — doesn't copy: the fork masks and renormalizes logits, which manufactures false confidence (a confused model prints 75%, and one test item was 98% confident on all three fields, all wrong). The data backs the skepticism: on 47 items, a 0.8 gate bounced 62% of traffic to the fallback, and Jev itself was 8 points overconfident on Banking77. 'Any LLM' holds only loosely — architecture decides whether you keep the single batched decode — and for stable labels, a 22M-parameter classifier beat Jev by 13 points in 8ms on a CPU.

deep-dives

Kai Explains

Published2026-09-14

Jev Mode in llama.cpp: Choose Instead of Write — Any LLM as a Millisecond Decision Model

Codacus adds 'Jev mode' to llama.cpp: a new /v1/decision endpoint on llama-server that takes instructions, a schema, and state, and returns every field with a value and a confidence score — in milliseconds. The trick is 'choose instead of write': instead of asking the model to write out JSON token by token (300+ passes for a 30-field object), you ask one question per field and read only the scores of the allowed option tokens, all in a single parallel pass over a cached context. Results: 17.3ms per decision in bulk on a 2GB model in 2.4GB VRAM, 306ms vs 3,415ms for the same 12-field JSON. The honest engineering caveats: attention-only models fly, hybrid/recurrent and MoE models hit batching walls, and a wrong pick is a wrong judgment — not a hallucination — so type safety isn't a substitute for knowing your task.

deep-dives

Codacus

Published2026-09-14

Needle 3: The 8MB On-Device Model That Beats 10× Larger Models

Needle 3 is Cactus Compute's third-generation on-device foundation model — a single 8-29MB binary that handles tool calling, structured extraction, and text embeddings entirely offline, with no cloud and no network. Its defining trick is the 'intelligence ladder': one set of weights where every depth from 2 to 20 layers is a complete, deployable model, so a developer picks a 2-layer version for a smartwatch or the full 20-layer for a flagship phone — all from one training run. This deep dive covers the architecture (Monarch Hadamard MLP, grouped-query attention, an engram n-gram memory that lets the 121M model do the arithmetic of a 50M one), the benchmark claims and their fine print — 'passes DeepSeek V4 Flash' means fine-tuned on DroidCall, not a general win — the Pebble smartwatch partnership, and the open-core business model. The honest verdict: a serious specialist for offline, act-on-voice use cases, not a general-purpose chat model.

deep-dives

Panda Making Money

Published2026-09-14

Needle 3: You Don't Need an LLM for Function Calling — A 35MB Automation Model

Needle 3, from Cactus Compute, is an automation foundation model for tiny devices — a ~35MB (53M-parameter) file that runs function calling on a CPU in ~66 milliseconds, no GPU, no JSON parser. It's a Simple Attention Network: a transformer with the feed-forward network replaced by a tiny MLP plus an n-gram embedding lookup, sliceable so any subset of its 20 layers becomes a smaller model for phones or microcontrollers. This deep dive covers the live 66ms home-automation demo, how it decomposes raw unstructured text directly into validated JSON schemas (no hallucination, no parser), the four gotchas that matter for real use — history leakage, default function values, messy input, and classification limits — plus its surprising secondary use as an embedding model. The honest framing: knowing where a model breaks matters more than where it shines.

deep-dives

Prompt Engineering

Published2026-09-14

DeepSeek V4.1-Flash: How a 552B Model Runs Like a 16B Model — The Causal Encoder-Decoder Redesign

DeepSeek V4.1-Flash has a 552-billion-parameter backbone but activates only 8B during prefill and 16B during decode — not by quantizing harder or just leaning on MoE, but by redesigning the transformer itself. The key move is a Causal Encoder-Decoder (CED): 20 encoder layers process the input and project the decoder's global KV cache once from the final encoder states, so the decoder never rebuilds the context. On top of that, Compressed Sparse Attention 2 with a hierarchical sparse indexer attacks long-context attention, FP4 KV compression plus SWA bounded replay cut the KV footprint to ~890 bytes/token, and DSpark speculative decoding, a 196B-parameter Engram conditional memory, and tunable reasoning (1-100) round out the runtime. The philosophical shift: instead of making the model smaller, make each token touch a tiny fraction of a much larger model.

deep-dives

AgenticEngineering

Published2026-09-14

Aerospace Is Probably The Best MacOS Tiling Manager I've Ever Used

AeroSpace is an i3-like tiling window manager for macOS built on a tree paradigm — and via exec-and-forget it doubles as an app launcher that replaces skhd. This deep dive covers Vim-key navigation, workspace emulation, the tree model for joining windows, fzf-based window switching, the ~/.config/aerospace/aerospace.toml config and custom modes, plus the SketchyBar + JankyBorders visual layer.

deep-dives

DevOps Toolbox

Published2026-09-11

The Cache Layer That Makes AI Agents 10x Faster?

LMCache reframes the KV cache as a shared, tiered memory layer for agent inference. This deep dive reconstructs the economics — 96.9% of agent context is reusable, ~95% of a 100k-token prefill is wasted work — and the benchmarks that show where it pays off (Google GKE: −79% TTFT at 100k tokens; Qwen3-235B-A22B: 3.98s → 0.29s via multi-process sharing) and where HBM still wins (AMD: 52 vs 25 requests). Plus the failure modes: PYTHONHASHSEED, unstable prefixes, and CacheBlend's reuse beyond prefix matches.

deep-dives

Code Unpacked

Published2026-09-11

Finally, The CORRECT Way to Run Local AI on a Mac

Samuel Gregory shares his definitive June 2026 verdict on running local LLMs on Mac: OMLX — an MLX-based inference server with persistent SSD-backed KV caching, two-tier RAM/SSD architecture, and seamless integration with coding agents like Claude Code, Pi, and Open Code.

deep-dives

Samuel Gregory

Published2026-06-30