💰 80 cents of every dollar 0:00
For every dollar spent on AI chips, Nvidia takes about eighty cents. That is the fact that explains the decade: Google, Amazon, Meta and now OpenAI build their own silicon, and newcomers keep raising billions to try. The question the video sets out to answer is not whether Nvidia should worry but something more useful — how are all these chips actually different?
The answer arrives through an unlikely protagonist. Jonathan Ross started Google's very first TPU as a twenty-percent project, left to found Groq and build the LPU, and in December 2025 Nvidia paid $20 billion to license the technology and hire him with it. His warning about the boom is the most quotable line in the video:
Baby turtles: hundreds hatch, a handful reach the sea. The framing matters because it is the same thing Google discovered in 2013 — that a brilliant chip design is the easy part, and everything hard happens between a working die and a running service.
🐢 Ross, the 20% project, and baby turtles 0:31
The video is built on two interviews — one with Ross, one with Abdel Sghiouar of Google Cloud — plus the primary literature: the 2017 TPU paper, Kung's 1982 systolic-array paper, and Nvidia's own documentation. That combination is why it can explain the same event at three levels: the business, the architecture, and the arithmetic.
| Protagonist | Chip | Core bet |
|---|---|---|
| Google (2013–2015) | TPU v1 | Strip the control logic, spend the silicon on a matrix multiply unit, feed it from on-chip SRAM |
| Nvidia (2017→) | Volta, Hopper, Blackwell, Rubin | Keep generality, add tensor cores, buy bandwidth with HBM and scale with NVLink |
| Groq (2016→) | LPU (originally TSP) | Delete external memory entirely; hold weights in SRAM and schedule every clock cycle ahead of time |
Everything else in the video is a consequence of those three positions. And, as it turns out, the two extremes ended up in the same rack — which is the real ending of the story.
📉 2013: the model Google couldn't deploy 1:09
The origin story is precise and worth retelling because it shows how a hardware decision gets made at a software company. In 2013 a small group at Google built a speech recognition system that cut the error rate by twenty percent — the biggest jump in twenty years — using a neural network with about 40 million parameters, not far from the 60 million in AlexNet, which had kicked off the deep-learning boom the year before.
Then they could not ship it. Google did not have the compute:
- Deployment was limited to the Nexus phone — not as a product decision, but because that was the only user base the available compute could serve.
- The projection said that if people used voice search for just three minutes a day, Google would have to double its data centres for this one feature.
🧮 Why the CPU is the wrong tool 2:08
A CPU is designed to do everything: run an operating system, handle network packets, respond to you. Its superpower is managing unpredictability — a program may branch on user input or read a memory address it only knows at runtime. To be fast at that, a CPU spends an enormous share of its silicon on control logic: predicting what happens next and recovering when the prediction is wrong.
Neural networks barely need any of it. Inference is one predictable operation repeated billions of times: matrix multiplication.
| Workload property | CPU strength | NN inference reality |
|---|---|---|
| Control flow | Branch prediction, speculation, out-of-order execution | Almost none — the same loop over a matrix |
| Data locality | Caches hide unpredictable access | Highly regular streaming of weights and activations |
| Precision needed | 32/64-bit floating point | Training in fp32, but inference often 8-bit integers |
So the CPU is not slow — it is spending its area on the wrong problem. Which raises the obvious next question: what about GPUs, which were already specialised?
🎮 Why the GPU was right, and where it fell short 2:46
GPUs had spent decades doing the same simple arithmetic to different slices of pixels: thousands of small cores instead of a handful of powerful ones. AlexNet had just been trained on two of them, and Google was openly buying off-the-shelf GPUs for training. In the 2010s training needed high-precision 32-bit floating point — exactly what GPUs were built for.
Inference is a different job, and two mismatches appear:
| Mismatch | Details |
|---|---|
| Arithmetic | Inference commonly compresses weights to lower precision, e.g. 8-bit integers — and at the time GPUs did not support integer operations well |
| Overhead | GPUs carried a lot of general-purpose machinery that inference never uses |
Then the decision, in Jeff Dean's analysis as Ross tells it: given what Google was going to spend and the capacity it needed, build an ASIC — an application-specific integrated circuit, designed for one narrow task: running neural networks and nothing else.
🔢 The ASIC decision and the MAC 3:42
Step one was to throw out all the predictive machinery and spend the space on a dedicated matrix multiply unit: a grid of 256×256 multiply-accumulate units — 65,536 MACs. A MAC is the atom of matrix maths: take two numbers, multiply, add to a running sum.
The two numbers are not equivalent, and the asymmetry is the key to everything that follows:
| Operand | What it is | Reuse pattern |
|---|---|---|
| Weight | Learned in training, then fixed forever | Same value used for every input the model will ever see |
| Activation | Intermediate result computed from the input | Used once, then discarded |
For a 70-billion-parameter model, every generated token requires roughly 70 billion MACs. Because TPUv1 targeted 8-bit integers, those MAC units were tiny — which is how 65,536 of them fit — and at peak the chip executed 92 trillion operations per second, about 35 times more than the CPUs in Google's servers.
🧱 SRAM, DRAM and the three-way trade 5:21
Ross states the designer's trilemma plainly: every choice is a trade between bandwidth, distance (how close the memory sits to the processing unit) and capacity.
| SRAM (on-chip) | DRAM (off-chip) | |
|---|---|---|
| Built from | Transistors — six per cell | Capacitors |
| Manufactured | Same process and die as the logic | Separately, then soldered onto the board |
| Area cost | Expensive: only so much fits | Cheap and 10–20× denser |
| TPUv1 numbers | 28 MiB | 8 GB of DDR3, about 10 cm away over wires |
| Latency | Deterministic, identical every time | Unpredictable — queueing, refresh, contention |
The video makes an important structural point: GPUs have exactly the same split. Nvidia's K80 — the GPU in Google's racks at the time — is two GPU dies on one card, each with its own DRAM soldered around it. Both architectures live with the same hierarchy; they differ in what they do about it.
⛔ The arithmetic that breaks: the von Neumann bottleneck 7:21
Before a model runs, its weights are loaded into DRAM. During inference they travel from DRAM into the on-chip matrix unit through the memory ports. On TPUv1 that channel carried 34 billion bytes per second.
Now the naive calculation the video does on screen, and it is the heart of the whole piece. Suppose each of the 65,000 MACs fetched its own fresh weight every clock cycle. At 700 MHz that is a demand of about 46 trillion bytes per second — three orders of magnitude more than the memory can deliver.
| Value | |
|---|---|
| Naive weight demand | ~46 TB/s |
| Available bandwidth | 34 GB/s |
| Ratio | ~1,350× |
| Consequence | Processor idle ≈99.9% of the time |
And this is not a 2010s deep-learning problem. It is as old as computing. In 1945 John von Neumann proposed separating a processor from a memory, with a wire between them — the design that powered almost every computer since. In 1977, accepting the Turing Award, John Backus named the consequence:
That is the title of the video, and it reframes the entire AI-chip boom: Nvidia, Google and Groq are not competing on transistors. They are competing on ways around a bottleneck identified half a century ago.
❤️ Systolic arrays: a heart that pumps data 9:14
The way around it was published in 1982 by H.T. Kung at Carnegie Mellon. In a von Neumann machine each operation costs one memory access: fetch, compute, fetch, compute. Kung's fix was to amortise the fetch — read a piece of data once, pass it through many processing elements, and store the result back once. The array of elements becomes a conveyor belt, with a new piece of data joining on every clock cycle.
The metaphor is the reason the design has its name. Kung imagined memory as a pulsating heart: in the systolic phase it contracts and pushes blood out; the blood flows through the organs as data flows through the processing elements.
How the wave does matrix multiplication
The TPU uses a variant called weight stationary, and the mechanics are worth following because they explain why the same weight can be reused without reloading:
- Weights stream in through the north edge, row by row, and then stay in place for the whole computation.
- Activations are transposed, then their rows offset by one cell each — a staircase — so each arrives at the right column at the right cycle.
- Every clock cycle the block of activations shifts one position right; each MAC multiplies the arriving activation by its stationary weight and adds a running sum coming from the north.
- Partial sums flow down through the south edge; the sums that accumulate at the bottom are exactly the output matrix.
The payoff is arithmetic. One weight, fetched once, is used once per activation in the batch — so weight traffic drops by a factor equal to the batch size. To run TPUv1 at peak, Google needed a batch of 250.
🔬 Two die floor plans, two philosophies 12:48
The clearest way to see the difference is to look at where the silicon goes.
| TPUv1 die | Share of area | Nvidia K80 die |
|---|---|---|
| Matrix multiply unit (systolic array) | ~25% | Dozens of identical streaming multiprocessors sharing a central cache |
| Activation buffer feeding it | ~33% | Each SM has its own instruction cache, warp schedulers, register files |
| Control logic | ~2% | A significant portion of the die for inference-irrelevant circuitry |
The TPU removed caches, branch prediction, out-of-order execution, multiprocessing and speculative prefetching. The minimalism reached the instruction set too: while Intel's x86 has about a thousand instructions, the first TPU barely supported a dozen — matrix multiply, plus basics like read weights or read host memory for inputs.
It does not even fetch its own instructions: an external host CPU sends them over PCIe into a local instruction buffer. In Google's own words, the first TPU was closer in spirit to a floating-point co-processor than to a GPU.
| Result against Nvidia's K80 (Google's ISCA 2017 paper) | Figure |
|---|---|
| Inference speed | 15–30× faster on average |
| Die-level short programs | ~15× faster than the K80 die, at less than half the power |
| Performance per watt | 30–80× better |
| MACs vs the K80 | 65,536 8-bit vs 2,496 32-bit — 25× more |
| On-chip memory | 28 MiB vs 8 MiB |
Nvidia would shrink that gap over the next few years — and the video is careful not to pretend the TPU ended the contest.
📊 Did it solve the bottleneck? No. 15:00
This is where most explainers stop flattering their subject, and this one does not. The TPU sped up every workload Google benchmarked — and yet four of the six production applications remained memory-bound, exactly as the 2017 paper reported. The bottleneck moved; it did not disappear.
Eight TPU generations have since chipped away at it, and Nvidia built its own line — Volta in 2017, then Hopper, Blackwell and now Rubin. Their differentiator is an on-chip Tensor Core dedicated to matrix multiplication. Nvidia has not disclosed the exact mechanism, but one meaningful difference is public: a TPU multiplies two large matrix tiles at once, while a GPU has many tensor cores each multiplying two very small tiles. That decomposition hands more power to the runtime scheduler, which decides which thread — and therefore which matrix slice — runs next.
| Lever | What happened | Why it matters |
|---|---|---|
| Memory bandwidth | TPU v1 → v8: ~200×, with the single biggest jump at v2 | This is the lever that actually attacks the bottleneck |
| HBM | v2 switched from board-soldered DDR3 to High Bandwidth Memory: DRAM dies stacked vertically on a silicon interposer a few microns from the processor | Shorter distance, thousands of wires, more bandwidth. Nvidia made the same switch before Volta |
| Peak compute | ~50× from v2 to v7 at bf16 | Smaller transistors, more of them, higher clocks — the familiar story |
| Scale-out | TPU v7 pods of 9,000+ chips; Nvidia's NVLink at 72 GPUs per rack | Necessary, and irrelevant to the per-device bottleneck |
And the bottleneck is still measurable in serving. In 2023 DeepMind served a quantized Llama 3 7B on a TPU v5e and still needed a batch size of 120 to keep the chip busy. A decade in, HBM remains a von Neumann bottleneck for both families. Which raises the question the second half of the video answers: what if you removed the HBM entirely?
⚡ Groq: delete the DRAM 17:51
Groq's answer is the most radical in the industry: no external memory at all. The weights live on the chip, in SRAM, which can be read roughly 10× faster than even the best HBM. To run a model larger than one chip, you spread the model across many chips and let the intermediate results move between them.
| Groq LPU (first generation) | Figure |
|---|---|
| On-chip SRAM per chip | 230 MB |
| External DRAM | None |
| Chips needed for a 70B model | ~576 |
| Llama 2 70B throughput (Feb 2024, ArtificialAnalysis) | 241 tokens/s — more than double other providers |
So the memory wait is gone and the price is paid in coordination: hundreds of chips that must exchange data constantly, in a fixed pattern, with no drift. Which is only possible because of a property of SRAM that has nothing to do with bandwidth.
📅 The bill for determinism 19:40
Ross explains the consequence with a calendar analogy: if all your meetings are fifteen minutes long, you can schedule them back to back, because you know exactly when each ends and the room frees up. Groq's compiler does this for every instruction, every memory access and every message between chips, down to the clock cycle.
Google compiles ahead of time too — XLA takes the model's computational graph and emits the instruction stream, and splits work across TPUs. But XLA only fixes the order of instructions, not their exact timing, "and it actually can't, because reads from DRAM take an unpredictable number of clock cycles". Before a matrix multiplication, the compiled program starts copying weights from DRAM and then waits on a flag that flips when the copy is done. Groq's compiler never has to insert that wait.
The results are striking: Groq's engineers predicted the latency of a BERT model before running it and were off by only 1%, and in production the same request takes exactly the same number of cycles every time. Reliability, bought with determinism.
🚚 Trucks, vans, and the KV cache 25:38
Before declaring a winner, the video does the honest arithmetic. Even for a modest Llama 2 70B you need 576 LPUs — "that's a lot of silicon holding weights instead of doing multiplications", at a reported $15,000–20,000 per chip. Ross's own framing of the trade is unusually fair to his competitor:
Which leads to the resolution, and the best engineering insight in the video: LLM inference is two different workloads.
| Stage | What happens | Natural hardware |
|---|---|---|
| Prefill | All prompt tokens processed in parallel — behaves like a big batch | GPU: high throughput, tolerates latency |
| Decode | Generates one token at a time, repeatedly reading all weights — memory-bound by construction | LPU: weights already on-chip, no HBM wait |
One wrinkle prevents a clean split. When generating a token, the model attends to every previous token, so it caches two activations per past token — a key and a value — in the KV cache. That cache grows with every token and every user, into gigabytes: far more than the LPU's SRAM can hold. So attention stays on the GPU, and the feed-forward layers — where most of the weights live — go to the LPU.
And in March 2026, at GTC, Nvidia showed exactly that division of labour: a rack of 256 Groq chips, now branded the LPX, designed to sit next to a rack of Rubin GPUs. If you can't beat them, buy the licence.
| Nvidia Groq 3 LPX (per rack, from Nvidia's own specs) | Figure |
|---|---|
| LPU chips | 256 |
| SRAM per chip / per rack | 500 MB / 128 GB |
| On-chip SRAM bandwidth per rack | 40 PB/s |
| Scale-up bandwidth per rack | 640 TB/s |
| FP8 inference compute per rack | 315 PFLOPS |
🛡 Nvidia's answer, and the baby turtle problem 27:36
On the TPU competition, Nvidia's public attitude is relaxed, and the argument is not about performance:
That is a real moat and a real constraint. Generality is why Nvidia could sell to every industry and why its software stack compounds; it is also why its chips carry silicon an inference-only workload never uses. Whether generality or specialisation wins is not a technical question — it depends on how much of the world's compute demand ends up being matrix multiplication at low precision, and on how quickly a specialised vendor can reach production scale.
The question the video leaves open is the right one: not "will Nvidia be replaced?" but "how many specialised architectures can the market support at production scale?" The LPX rack is a partial answer — the incumbent absorbing the specialist rather than losing to it.
✅ Claims checked
| Claim | Verdict | Evidence |
|---|---|---|
| Nvidia paid $20 billion to license Groq's technology and hire Jonathan Ross | Correct | Non-exclusive licensing agreement announced 24 December 2025; Ross, Sunny Madra and other staff joined Nvidia; Groq remains independent under CEO Simon Edwards (Groq newsroom, CNBC) |
| Ross started Google's first TPU as a 20% project | Consistent | Confirmed in Ross's own account and in his later role founding Groq; TPUv1 deployed in Google datacenters from 2015 |
| 2013 speech model: ~40M parameters, 20% error-rate cut, Nexus-only deployment | Speaker-reported | Matches the well-known Google account of the TPU's origin; specific figures are not in the 2017 paper |
| Three minutes of daily voice search would have doubled Google's data centres | Widely reported projection | Recounted by the TPU team; an internal projection, not an audited number |
| TPUv1: 256×256 MACs, 92 TOPS, 28 MiB on-chip, 8 GB DDR3, 34 GB/s, 700 MHz | Correct | ISCA 2017 paper (arXiv 1704.04760): 65,536 8-bit MACs, 92 TOPS, 28 MiB, 34 GB/s, 8 GiB DDR3 |
| TPUv1 ~15× faster than the K80 at less than half the power | Correct | Paper: ~15× faster than the K80 die running short programs, using less than half the K80's power; 15–30× overall, 30–80× better perf/W |
| Naive weight fetch would demand ~46 TB/s against 34 GB/s | Arithmetic checks out | 65,536 MACs × 2 bytes × 700 MHz ≈ 92 TB/s of operand traffic; the ~46 TB/s figure assumes one 1-byte weight per MAC per cycle |
| Backus coined "von Neumann bottleneck" in his 1977 Turing Award lecture | Correct | Backus, "Can Programming Be Liberated from the von Neumann Style?", 1977 |
| Systolic arrays proposed by H.T. Kung, CMU, 1982 | Correct | Kung, "Why Systolic Architectures?", IEEE Computer, 1982 |
| Four of six benchmarked TPU workloads remained memory-bound | Correct | ISCA 2017 paper: four of the six production NN applications were memory-bandwidth limited |
| Batch size 250 to reach TPUv1 peak | Correct | Paper reports batch 250 for the MLP workload achieving ~86% of peak |
| TPU v1→v8: ~200× bandwidth; v2→v7: ~50× compute at bf16 | Company figures | Consistent with Google's published generation comparisons; not independently audited here |
| Groq LPU: 230 MB SRAM, no external memory, ~576 chips for 70B | Correct | 230 MB SRAM per GroqChip; ~576 LPUs for a 70B model (SemiAnalysis, Groq-compatible estimates) |
| Llama 2 70B at ~240 tokens/s, about twice the runner-up | Correct | ArtificialAnalysis, Feb 2024: 241 tokens/s, "more than double the speed of other hosting providers" |
| Groq predicted BERT latency within 1% before running it | Consistent | From Groq's 2022 determinism work; the transcript's "bird model" is BERT |
| LPU cost $15–20k per chip | Speaker-reported | Order-of-magnitude figure for early GroqChips; not a published list price |
| GTC March 2026: 256-chip LPX rack beside Rubin GPUs | Correct | Nvidia Groq 3 LPX: 256 LPUs, 128 GB SRAM, 40 PB/s SRAM bandwidth, 640 TB/s scale-up, 315 PFLOPS FP8 |
| Attention stays on the GPU, feed-forward on the LPU | Correct | Nvidia describes prefill and part of decode on Vera Rubin, feed-forward offloaded to LPX — attention/FFN disaggregation |
| Cerebras went public in 2026 and supplies OpenAI | Correct | Nasdaq listing (CBRS) in May 2026; OpenAI compute agreement valued at over $20B |
Transcript notes. Auto-captions garbled many proper nouns: "Grock" → Groq, "Wonman / vonoman / Foyman" → von Neumann, "Bakus" → Backus, "HD Kong" → H.T. Kung, "Reuben" → Rubin, "Volulta" → Volta, "KD cache" → KV cache, "Llama 37B" → Llama 3 7B, "Llama 270B" → Llama 2 70B, "bird model" → BERT. Where a garble could change meaning, the corrected term is used and noted here.
🔑 Key takeaways
- Every AI chip is an answer to one 1945 question: how do you feed a processor from memory that is too slow, too far, or too small. GPUs, TPUs and LPUs differ in which part of that trade they sacrifice.
- CPUs spend their silicon on unpredictability. Inference is a predictable loop over a matrix, so control logic — branch prediction, speculation, out-of-order execution — is 2% of a TPU die and a large share of a GPU's.
- A memory bandwidth number decides everything. 34 GB/s against a 46 TB/s naive demand means 99.9% idle. Compute is not the scarce resource; data movement is.
- Systolic arrays are a 1982 idea doing the work in 2026. Kung's conveyor belt amortises memory access, and the TPU is his array grown from a vector into a grid.
- Weight-stationary reuse is the trick: load weights once, stream activations through, and watch your memory traffic drop by the batch size — at the cost of latency, because a batch of 250 has to be gathered first.
- The bottleneck survived the revolution. Four of six TPU workloads were still memory-bound after the ASIC, and HBM keeps both families on the same treadmill.
- SRAM's real advantage is predictability. A read takes the same number of cycles every time, which is what lets a compiler schedule every instruction and every inter-chip message ahead of time.
- Determinism is not free: model-specific compilation, recompiles when parameter counts change, bucketing to fix tensor shapes, and clock re-synchronisation between chips.
- Inference is two workloads. Prefill is throughput-shaped and belongs on GPUs; decode is memory-bound and belongs where the weights already live. The architecture that wins is the pairing, not either chip.
- The KV cache is why attention can't leave the GPU. It grows with tokens and users into gigabytes — more than any SRAM-only design can hold.
- Nvidia bought the specialist instead of beating it. A $20B licence became a 256-chip LPX rack co-designed to sit beside Rubin: heterogeneity inside one vendor's platform.
- Designing a chip is easy; production is the filter. The baby-turtle problem is compilers, memory systems, networking, reliability and customers willing to bet a data centre — which is why capital alone does not create a competitor.
🔗 Resources
- The source video — Julia Turc, "It's von Neumann's fault | GPU · TPU · LPU", 29:48, 2 November 2026
- In-Datacenter Performance Analysis of a Tensor Processing Unit — Jouppi et al., ISCA 2017, the primary source for every TPUv1 figure
- Why Systolic Architectures? — H.T. Kung, IEEE Computer, 1982
- Can Programming Be Liberated from the von Neumann Style? — John Backus, 1977 Turing Award lecture
- NVIDIA Groq 3 LPX — official specification for the 256-chip inference rack
- Inside NVIDIA Groq 3 LPX — technical blog on determinism, SRAM bandwidth and scale-up
- Groq–Nvidia licensing announcement — 24 December 2025
- Full interview with Jonathan Ross — the source of the LPU quotes
- Full interview with Abdel Sghiouar — Google Cloud, on TPU design choices
- The memory bottleneck for LLMs — companion video on the same theme