1 The Four Pillars of Modern AI
Pourya opens by framing DeepMind's history: Demis Hassabis has always treated AGI as a long-term scientific mission to recreate the human mind, not a race to build a profitable software product. When Go was considered unsolvable for AI, DeepMind invested heavily with no immediate return. Now, history is repeating itself — DeepMind is pouring resources into another problem nobody else wants to touch.
Modern AI is held up by four main pillars:
- Transformer-based — the attention architecture
- Autoregressive — next-token prediction
- Pre-trained — large-scale training on existing data
- Generative — producing outputs (text, images, video)
Most researchers agree this specific combination has enormous potential but also theoretical limitations that prevent it from reaching human-level intelligence. DeepMind is already moving beyond all four.
2 Replacing the Transformer
The transformer's core mechanism — each token looking at all other tokens — is powerful but doesn't scale cheaply. Interactions grow quadratically with sequence length. While modern systems use efficient approximations (sparse patterns, hybrid architectures), most leading labs have avoided experimenting with entirely new foundations.
DeepMind is actively exploring alternatives through multiple research lines:
- Titans — "learning to memorize at test time." Uses a surprise mechanism to selectively retain unexpected information, improving common-sense reasoning, genomics, and time-series tasks
- Nested Learning and Titans + Miras — helping AI develop long-term memory
- Griffin architecture — mixing gated linear recurrences with local attention
- RecurrentGemma — moving past transformers for efficient open language models
- Gemma 4 — maximum compute and memory efficiency
3 The Griffin Architecture & RecurrentGemma
Pourya uses a brilliant book-reading analogy to explain the difference between standard attention and Griffin:
- Standard transformer — reading page 100 while carrying all 99 previous pages in front of you (the KV cache). The stack becomes an unbearable liability
- Griffin — instead of carrying everything, you maintain an index card that is constantly rewritten for every page. Local attention handles only ~2,000 tokens for the current "page"
The key innovations in Griffin:
- Gate weights — matrices that learn during training which information is worth remembering (bold words, specific numbers, chapter starts, and far more complex strategies)
- Recurrence gate — integrates past information with new information into a unified representation. If the main character is described as tall on page 1 and blue-eyed on page 65, the index card contains a unified representation — not a list
RecurrentGemma showed jumps on benchmarks, especially for long context — one of the rare transformer challenges given a serious shot at proving itself.
4 Gemma 4: Edge Efficiency at Scale
DeepMind squeezed the results of these experiments into Gemma 4 — a model designed for maximum compute and memory efficiency:
- Sparsity via MoE — the "4 billion parameter" model is actually ~25B parameters, but only ~4B are active at any time using Mixture of Experts and per-layer embedding methods
- Multimodal — accepts raw video and audio on edge devices
- 256K token context — achieved through a hybrid of local sliding window attention (some layers look at ~1,000 tokens) and global attention (other layers see everything)
The analogy: reading a paragraph with intense local focus, then going back to read relevant pages to understand it in the full context of the book.
5 Beyond Autoregression
A critical misconception: many assume that the transformer is inherently a next-token predictor. But the original transformer was invented for translation and wasn't purely autoregressive. It was Alec Radford who turned it into a decoder-only autoregressive model for general self-supervised learning.
The main invention of the transformer — self-attention — doesn't require next-token prediction at all. This opens up a whole area of transformer-based but not autoregressive models.
Professor Stefano Ermon (pioneer of diffusion models) explains: autoregressive models generate text one token at a time, left to right. Diffusion models generate by iterative refinement — starting with a rough guess and refining it. Crucially, this process is highly parallel: the network can modify multiple tokens simultaneously.
6 Diffusion Language Models: Three Advantages
Pourya breaks down three key advantages of diffusion LLMs over autoregressive models:
- Speed and efficiency — diffusion models need far fewer iterations (tens to low hundreds) vs. thousands or tens of thousands for autoregressive. Already ~10x faster in practice because GPUs love parallelism
- Smarter outputs — autoregressive models can't revise a token once generated. Diffusion models correct mistakes by working on the entire response holistically
- Flexibility — autoregressive models expect the prompt as a prefix. Diffusion LLMs allow the prompt at any arbitrary position. For code, this is transformative: the model naturally sees the entire codebase and completes the missing part, instead of awkwardly rewriting parts it shouldn't touch
Inception Labs (a small Stanford spinoff) has already introduced Mercury — the first reasoning LLM powered by diffusion, ridiculously faster than models of its size while being competitive in quality. Google DeepMind introduced Gemini Diffusion in mid-2025 — the only major lab working on a diffusion language model.
Meanwhile, OpenAI recently shut down Sora 2 (a diffusion model), letting go of a billion-dollar Disney contract to double down on autoregressive GPT technology. Sam Altman described diffusion as "a different branch of the tech tree" that's too hard to pursue simultaneously.
7 The Trillion-Dollar Question: Why Video Matters for AGI
Why would DeepMind invest so heavily in realistic video generation? Because this "slot machine" may be the clearest path toward AGI.
Video diffusion models don't predict raw pixels. Instead:
- Encoder — compresses video into latent space, building a compact "visual language" that preserves underlying regularities of the real world
- Diffusion model — learns to write in this compressed visual language inside the frozen latent space
- Decoder — reconstructs pixels from the latent representation
The key insight: a cat walking across a room isn't just any possible image. Physics, geometry, object permanence, lighting, and motion continuity massively reduce the space of plausible futures. Real-world videos occupy an extremely tiny, structured subset of all possible frames.
8 Jagged Intelligence & The Language Limitation
The biggest flaw in current LLMs: jagged intelligence. Models can win math olympiad gold medals but still give absurd real-world advice. Other labs are publicly accepting this jaggedness, focusing on building narrow but useful tools.
But if the goal is AGI, this must be solved. Written language is just an artificial artifact — a super lossy compression of our inner model of the world. It's not a fundamental part of human intelligence.
Pourya's analogy: someone who's read a thousand books on Formula 1 racing vs. Lewis Hamilton. The reader knows all the facts and physics, but the driver has developed such a deep understanding that the car feels like an extension of his body. The human mind relies on a wordless internal representation of reality — that's the core of intelligence. Generating words to approximate it is the trivial part.
9 World Models: Hassabis vs LeCun (JEPA)
This section covers what Pourya calls "the most fascinating argument at the center of AI right now" — a defining disagreement people will write books about a decade from now.
Hassabis (DeepMind): Generative World Models
Demis argues that video generators like VEO are steps toward a world model — a system that understands the mechanics and causality of the world. A video model generating 20 seconds of realistic physics is exhibiting understanding of "intuitive physics." This would enable AGI to plan long-term in the real world.
LeCun (Meta): JEPA — Joint Embedding Predictive Architecture
Yann LeCun argues that generative models waste compute predicting pixel-level noise — random leaf movements, for instance. His alternative, JEPA, doesn't generate at all. Instead:
- A context encoder sees the visible part of input
- A target encoder processes the whole input
- A predictor predicts the target representation from context — only matching representations in embedding space, never reconstructing pixels
If a ball rolls behind a wall, JEPA predicts a compatible future representation where the ball emerges on the other side — developing implicit understanding of object permanence without ever generating the video.
The Nuanced Split
Both architectures learn world models, but the key difference:
- Generative — keep enough information to reconstruct the original input
- JEPA — predict a compatible representation of the missing part
Theoretically, JEPA should eventually outperform generative systems. Empirically, generative models are still performing much better across many important domains.
10 Continual Learning vs Large-Scale Pretraining
Highly respected researchers like Ilya Sutskever and Richard Sutton suggest that large-scale pretraining is probably a dead end. Their argument has two parts:
1. Learning from Experience
The physical world is messy — full of unorganized goals, subtle learning signals, and dynamic interactions. Training in an isolated lab with rigid goals will never result in a truly intelligent system. The model should need only minimal pretraining paired with an extremely efficient learning algorithm.
2. Biological Plausibility
Ilya's key point: pretraining "overshot the target." A human being is not an AGI — humans lack huge amounts of knowledge. Instead, we rely on continual learning. Superintelligence isn't a finished mind that knows how to do every job — it's "a mind which can learn to do every single job."
Pourya illustrates: you don't need to crash a car to understand danger. The moment you sit behind the wheel, you already have a sense of balance, control, and whether something feels wrong. The brain constantly evaluates itself long before any catastrophic outcome because learning signals are so rich.
Hassabis disagrees — he argues the quickest path to AGI is to ingest all existing world knowledge as a prior, then build on top. His betting: the final AGI system will have large multimodal models as part of the solution, but they won't be enough on their own. You'll need additional planning and search on top.
11 DeepMind's Grand Bet
The video reveals DeepMind's actual endgame — a three-layer AGI architecture:
- Layer 1: Generative diffusion models — build hyperrealistic, physics-grounded simulations of the world (VEO, Genie, Gemini Diffusion)
- Layer 2: Multimodal autoregressive core — train Gemini inside these simulated worlds to develop an explicit, evolving model of reality internally. This is Gemini Omni — not just another video generator, but an early iteration of DeepMind's world model
- Layer 3: Planning and search — the AlphaGo-style reasoning layer on top
From one angle, DeepMind is doubling down on generative AI and massive pretrained models. From another, they're actively pushing research far beyond transformers and autoregression. It's a highly specific bet on exactly what AGI should look like — and it will only take a couple more years to see who's right.
🎯 Key Takeaways
🔑 Key Takeaways
- DeepMind is challenging all four pillars of modern AI simultaneously — transformers, autoregression, generation, and pretraining — while everyone else doubles down on them
- Titans introduces selective memory — a "surprise mechanism" that only remembers unexpected things, enabling adaptation at inference time and a path toward continual learning
- Griffin replaces the KV cache with an evolving "index card" — gated linear recurrences that learn what's worth remembering, paired with local-only attention
- Gemma 4 packs 25B parameters into a 4B active footprint — running multimodal (video + audio) on edge devices with 256K context via hybrid local/global attention
- Diffusion LLMs are ~10x faster than autoregressive models, can revise outputs holistically, and handle code infill naturally — Inception Labs' Mercury already proves the concept
- OpenAI shut down Sora 2 to focus on GPT — the opposite of DeepMind's multi-branch strategy. DeepMind maintains VEO, Genie, Gemini Diffusion, Imagine, and more
- Language is a lossy compression of intelligence — the real core is a wordless internal world model. Video generation forces AI to learn the underlying physics
- Hassabis vs LeCun is the defining debate — generative world models (reconstruct reality) vs JEPA (predict compatible representations). Generative wins empirically; JEPA may win theoretically
- Sutskever argues pretraining "overshot the target" — AGI isn't a mind that knows everything, it's a mind that can learn anything. Continual learning, not data scale, is the key
- DeepMind's actual endgame — diffusion-based world simulators → Gemini trained inside them → AlphaGo-style planning on top. Gemini Omni is the early iteration of this vision
🔗 Resources & Links
- RecurrentGemma — DeepMind's recurrent architecture for efficient open language models
- Griffin Architecture Paper — mixing gated linear recurrences with local attention
- Gemma 4 Model Card — edge-efficient multimodal model with 256K context
- Titans Architecture Paper — learning to memorize at test time
- Nested Learning Paper — extending Titans with hierarchical memory
- Mercury by Inception Labs — first reasoning LLM powered by diffusion
- Gemini Omni — DeepMind's multimodal autoregressive world model
- V-JEPA by Meta — Yann LeCun's video Joint Embedding Predictive Architecture