The Question That Inspired LoRA: Intrinsic Dimension and Why Tiny Ranks Work

0:00Which task is harder for a neural net to learn — recognizing a handwritten digit, or teaching a simulated 3D humanoid (13 body parts, 17 motors) to walk? In 2018 researchers found a way to put a number on task difficulty, and it's the same number that explains why LoRA — today's most popular fine-tuning method — works so efficiently.

Why fine-tune at all

1:08The elephant in the room: with capable LLMs, why not just describe the task in a prompt? The motivating example is Cursor's Auto mode, whose router sends each request to the cheapest model that can handle it. The router could use an off-the-shelf LLM as a judge, but that judge must be cheap — and small models are unreliable at absolute scores. A study showed Qwen 3.8B changes 35% of its answers if you list scores in arbitrary order, and 46% in descending order. Prompt quirks have outsized effects. Fine-tuning is "show, don't tell": you feed thousands of queries with ground-truth scores, and the model learns what a "2" or "4" means from examples. In-context learning works too, but you pay for those example tokens on every query; fine-tuning pays once. Cursor chose fine-tuning — training its router on 600,000+ live requests.

What full fine-tuning costs

3:22Full fine-tuning updates every weight and stores gradients for each one, every step — plus optimizer state (Adam/Muon) that doubles or triples the space. For Kimi K3 (2.8 trillion parameters), the adaptation Δ alone has 2.8 trillion entries, and full fine-tuning needs at least ~30 TB of GPU memory, not even counting activations. That's the problem parameter-efficient fine-tuning (PEFT) solves.

How LoRA works

4:57LoRA (Low-Rank Adaptation) constrains the update to each weight matrix W to a product of two thin matrices, B × A, with an intermediate rank r — so a single M×N matrix (up to 242M values on one layer of Kimi K3) shrinks to r×M + N values, under 2% of the original. B and A are the only learnable parameters; the pretrained W₀ stays frozen. If you just need the job done, treat r as a hyperparameter and keep the smallest value that holds accuracy — sometimes r = 1 works.

The geometry

6:36The intuition: full fine-tuning lets the "ball" roll downhill in any gradient direction — the whole plane. LoRA restricts the search to a set of directions. A single line extends to infinity (a direction u you move along by amount α), so a line is a smart restriction — it's how a valley-shaped landscape can be reached even when you don't know where the minimum is. Rewriting B×A as a sum of outer products: the rows of A are the allowed directions, the columns of B are the coefficients (how much to move along each). Rank r is the number of linearly independent directions, and setting a low r constrains Δ to low rank. In 2D, allowing more than one direction stops being restrictive; with 2.8 trillion parameters, even thousands of directions cover a tiny region.

Intrinsic dimension

10:09The backstory. In 2018, a paper from Uber AI (Li et al., arXiv 1804.08838) asked: if you fix a set of random directions and train only their coefficients, how many directions do you need to reach 90% of a freely-trained model's performance? That count is the task's intrinsic dimension. Their results: balancing an inverted pendulum = 4, CIFAR-10 ≈ 9,000, Atari Pong = 6,000, ImageNet > 500,000 (they ran out of compute) — and the opening question's answer: MNIST ≈ 750 and the walking humanoid ≈ 700, about equally hard ("perhaps vision is harder than we think because we do it so well"). In 2020, Facebook AI (Aghajanyan et al., arXiv 2012.13255) ran the same on a pretrained model: RoBERTa-large (355M params), fine-tuned for paraphrase classification, had an intrinsic dimension of 322 — a millionth of the model size. A few hundred directions adapted a huge model. The LoRA authors saw the practical potential: if a few random directions work, could a few learned directions do better?

The demo

15:21She builds the query router: fine-tune Qwen3.8-27B (small enough to be cheap, big enough to be serious) on a prompt-complexity dataset (~50% of prompts score "2", so a naive always-predict-2 baseline hits 50%). Baselines: prompting with an ascending guide = 26%, descending = 48.5% (her theory: recency bias toward the most common label), in-context learning (2 examples/score) = 59.5% — a clear win, but every request becomes 2.5× longer and costs tokens each query. Then LoRA at r = 1 and r = 16, via Fireworks' serverless training API (~$2–3 per run on a 27B model): both climb from 48% to the mid-70s, best checkpoint 76% — above every baseline — and, crucially, no meaningful difference between r = 1 and r = 16. 19:12To see why, she measures the stable rank (via SVD) across all 448 adaptation matrices: most fall between 1 and 7, median 3 — the first direction carries a third of the energy. r = 16 is unnecessarily large, which is the intuition she set out to reach: the rank can be tiny because the task itself lives in a low-dimensional subspace.

Claims checked

Sources & further reading