Andrej Karpathy: Software 3.0, Transformers & the Full AI Engineering Stack (Stanford)

👋 The Hacker & Why Now

Karpathy opens with his trajectory: Stanford PhD student ~7 years ago, then OpenAI, then Tesla (Autopilot), and back to OpenAI "as of one week ago." At Stanford he worked on early neural networks connecting images and language (early CLIP-style models and image captioners); at OpenAI on generative image models and RL; at Tesla on the neural networks behind the Autopilot visualization. 0:00

"I suspect the reason I was invited to give a keynote is not any of that stuff, but the fact that I love to hack." He rattles off his side projects: convnet.js (a JS neural network library — "I did it for the lols"), being the reference human for ImageNet (he personally classified images into 1,000 categories including ~200 dog breeds for a week — "when you see human accuracy quoted on ImageNet, that's me"), activity-tracking apps, arxiv-sanity (paper recommender), blogging, and now YouTube videos + nanoGPT. 1:00

"There has never been a more interesting time to hack than today. Programming is changing very rapidly, and you are all explorers looking at the new vistas." 2:32

🖥️ Software 1.0 — 70 Years of Instructions

What is programming? "Some of you think of writing C++ code, giving instructions to a computer. Maybe Donald Knuth and the Art of Computer Programming. This is programming as it was for the last ~70 years — unchanged on a high level: spelling out instructions and designing an algorithm." 3:09

It got us very far — Linux is a diagram of "a very complicated software engineering project with lots of moving pieces, profilers and debuggers." "But not quite all the way. We started to see the cracks when we got to problems like image recognition." 3:55

The cracks: "You can't write an algorithm to recognize a cat in an image, because the cat can take on many different forms. You can't write a good chess engine by giving explicit instructions. You can't write an autopilot that way. And we probably are not going to build AGI by spelling it out for a computer." 4:15

🧠 Software 2.0 — The Data Engine

The new programming paradigm is neural networks — "not just another classifier in competition with a random forest. Neural networks are a new programming stack, and you program them slightly differently." 4:48

The Software 2.0 stack: you program by accumulating datasets and iterating on them — a "data engine." You then compile your dataset into a binary, where the compilation is neural network training and the binary is the weights. "This is the final program, written in weights — and you can't write it by hand. It comes out of optimization based on your dataset." 5:03

The data engine cycle (about 5 years of his Tesla life): start with a dataset → train a net → deploy → telemetry/monitoring → collect more data the network finds troubling → label it → some into test sets, some back into training → repeat. "You spin the cycle over and over again." 5:20

Crucially, 2.0 doesn't replace 1.0 — "they layer on top of each other. You still need a ton of 1.0 code to compile your software 2.0." Examples: computer vision became one big convnet; chess became a reinforcement-learning problem (reward 1 for a win, 0 for a draw, -1 for a loss); speech became a big net trained on tons of data (Whisper). 6:06

📚 LLMs — Next-Word Prediction Magic

"Over the last 2–3 years we're again in the middle of another transition." The story begins with large language models: they just predict the next word in a sequence, but with a trillion parameters trained on the whole internet, "something magical starts to happen." 7:04

Generation is just repeated prediction — predict the next thing, plug it back in, repeat — producing poems and arbitrary text. But more interestingly, you can use them to perform tasks: give context, then a few question-answer examples (few-shot, from the GPT-3 paper), and the model "takes on the task of giving the actual answer." 7:58

The juggler prompt: "A juggler can juggle 16 balls. Half are golf balls, and half of the golf balls are blue. How many blue golf balls are there?" Asked naively, it says 8 (wrong). But "let's think step by step" breaks the problem into smaller steps — "it has more tokens, more time to think, and a higher chance of getting it right." Accuracy jumped from 17% to 78.7%. Even better: "Let's work this out in a step-by-step way to be sure we have the right answer" hit 82%. 8:35

You're conditioning a next-word predictor: "Ask ChatGPT why it rains and it imitates the average answer on the internet. There are many people of different IQs describing why it rains — if you condition on 'the IQ-200 person should tell me,' you get a much better answer. You really have to narrow in on the slice of the prediction you want; otherwise it imitates the average case." 10:26

🎯 Prompt Engineering Is Real

Prompt design goes a long way, and it's now a legitimate job. Karpathy points to Riley Goodside — "one of the first prompt engineers I'm aware of, currently staff prompt engineer at Scale — just extremely good at all these prompts and techniques, and very helpful to me personally." 18:45

The famous framing tweets: "If previous neural nets are special-purpose computers designed for a specific task, GPTs are a general-purpose computer reconfigurable at runtime to run natural-language programs. The programs are specified in prompts, and GPT runs the program by completing the document." And: "The hottest new programming language is English." 19:50

🧪 The Model as Simulator

"GPT is kind of like a simulator — you can condition it into arbitrary universes and get really cool outputs." A series of examples: 11:22

🐧 Linux terminal simulation — "Act as a Linux terminal. I'll type commands, you reply with what the terminal should show." It hallucinates an entire filesystem — ls, cd, creating a jokes.txt, even running Python programs correctly "in the mind of the model." 11:35
🌐 Fake network — ping bbc.com produces a plausible ping with an IP that doesn't actually exist ("I double-checked, it's totally made up — but it looks like ~24.9 ms latency"). Even a curl POST to ChatGPT returns plausible JSON. 13:00
🏠 The smart-home brain — someone prompted GPT-3 to "pretend to be the sentient brain of a smart home," declaring appliances and rooms in plain English, outputting JSON commands. "We've programmed a smart assistant just by giving it text. That's pretty incredible." 14:10
⚙️ GPT-as-backend — the hackathon winner "GPT is all you need for backend": no Python, just an LLM that takes app state as JSON, applies English-described operations ("delete the last two todos"), and returns the new JSON state. 16:20
🔍 The Sydney prompt — the (allegedly leaked) prompt behind Bing's Sydney persona: "you can see how engineers at Microsoft programmed Sydney — it identifies as Bing Search, introduces itself a certain way, its output format, limitations, and safety rules, all described in English." 17:50

🇬🇧 Software 3.0 — English as the Language

Bringing the paradigms together: "Software 1.0 was the realm of 'I designed the algorithm' — with us for 70 years. Software 2.0 is dataset iteration — you design the dataset. Software 3.0 is you design the prompt — conditioning a large language model to perform tasks." 20:26

A thought that hit him "at random": "Software through prompting is also how you program humans. If you want humans to do something, you do it via prompt. It's interesting that our technology is converging to humans in this way." 20:44

The summary slide: all the programming languages didn't change the paradigm — "what changed the paradigm are neural networks, the data engine, and now the hottest language is English." And the practical tip: "if you want to use any of this in your hacks, the best way to get started is the OpenAI APIs. I don't say that because I work there — I work there because I say that." 21:20

📜 Transformer History

Karpathy traces the transformer's lineage with the framing of an experienced CV researcher who watched the field converge: 33:50

Year	Milestone
2003	Bengio et al. — first popular neural language model: a multi-layer perceptron predicting the 4th word from 3 words
2012	AlexNet — scale a big net on a big dataset, "and the recipe copy-pasted across many areas of AI"
2014	Sequence-to-sequence — LSTM encoder/decoder for translation, but with the "encoder bottleneck" (whole sentence squeezed into one vector)
2015	Bahdanau et al. — attention: "soft search" over source words to break the bottleneck
2017	Attention Is All You Need — "delete everything, keep attention"

The pre-2012 era: computer vision pipelines were "a zoo of kitchen-sink feature descriptors — sparse histograms, SIFT, color histograms, textons, tiny images, GIST — everyone had their favorite, and you put an SVM on top. It also didn't work. Today you'd be looking for a bug; back then you'd shrug." Worse, every field had its own vocabulary (POS tagging, morphological analysis, syntactic parsing, coreference resolution in NLP) so you couldn't even read papers across areas. 30:30

2012 changed it: neural nets "pop up everywhere — vision, NLP, speech, translation, RL — the same modeling toolkit." Then 2017: "the architectures converge to literally one architecture you copy-paste across everything." What changes is the details of the data and how you chunk and feed it in. 33:00

The brain hypothesis: "This is a hint we're converging to something the brain is doing. The cortex is homogeneous across the entire sheet — your auditory cortex and visual cortex look very similar; maybe the differences are like hyperparameters of a transformer. Maybe we're converging to a uniform, powerful learning algorithm." 33:57

The origin of "attention" — a translation exercise: Karpathy emailed Dimitri (Bahdanau) asking where soft attention came from. The reply: he wanted the decoder RNN to "learn to search where to put the cursor" in the source — inspired by English translation exercises in middle school, "your gaze shifts back and forth between source and target as you translate." "He's not a native English speaker, and that gave him an edge in machine translation that led to attention, then the transformer." The name "RNN-search" was deemed lame; Yoshua Bengio suggested "attention" — "so maybe Attention Is All You Need would have been called 'RNN Search Is All You Need.'" 37:40

Attention Is All You Need: "usually papers are incremental, adding one thing. This was a mix of multiple things combined in a unique way, and achieved a very good local minimum in architecture space." It deleted the RNN, kept attention, added positional encoding (attention has no notion of space), adopted the residual structure from ResNet, interspersed attention with MLPs, used layer norms, introduced multi-head attention, and set hyperparameters (4× MLP expansion) "that to this day are used." The only change that stuck since: pre-norm (layer norm moved before attention/MLP). "The GPTs you see today are basically the 2017 architecture." 39:20

In-context learning is the real magic: "I would have renamed 'Language Models are Few-Shot Learners' to 'Transformers are capable of in-context learning or meta-learning.'" With more examples in the prompt, accuracy improves — without any gradient descent. "The transformer is learning in its activations as it reads the sequence — an inner loop of in-context learning, vs. the outer loop of SGD training." Recent papers (e.g. the "raw operator") hint that transformers may implement something like gradient-based learning internally. 41:17

🔗 Attention = Message Passing

Karpathy's own interpretation, deliberately different from standard presentations: "To me, attention is the communication phase of the transformer. The transformer interleaves two phases — communication (multi-headed attention) and computation (the MLP)." 44:25

Forget machine translation — think directed graphs: each node stores a vector; it emits a query (what I'm looking for), a key (what I have), and a value (what I'll communicate), each via a linear transform. Nodes broadcast keys; your query dot-products against incoming keys to get "interestingness," softmax normalizes it, and the weighted sum of values flows to you and updates you. "This is happening for each node, then we update at the end." 45:10

Term	Meaning
Multi-headed attention	Attention applied multiple times in parallel, different Q/K/V weights — "heads = copy-paste in parallel"
Layers	"Copy-paste but in series"
Self-attention	Keys and values produced from this node itself
Cross-attention	Queries from this node, but keys/values from an external source (the encoder)

"Self-attention and cross-attention only differ in where the keys and values come from — algorithmically it's the same operations." In the encoder-decoder graph, encoder tokens are fully connected to each other; decoder tokens connect to the encoder and to everything before them (triangular structure), because future tokens "give away the answer." 49:00

💻 nanoGPT Walkthrough

Karpathy walks through nanoGPT, his ~300-line minimal GPT-2 reproduction trained on OpenWebText (one node of TPUs for 38 hours). The toy dataset is tiny-Shakespeare — all of Shakespeare concatenated into a 1 MB file. 52:10

Tokenization: transformers can't take raw text — "in the simplest case, every character gets an integer." (Real models use BPE / sentencepiece / wordpiece tokens.) Multiple documents are separated by special end-of-text tokens. 53:40

Batching: take chunks of the 1-D sequence. Block size = max context length; batch size = how many sequences in parallel (to saturate the GPU). "In a single 4×8 batch, every position is both an input and a target — the real batch size is more like B×T." 54:50

Forward pass: index each token into an embedding table → add positional encodings ("the 'what' and 'where' combined additively") → feed through a series of transformer blocks → layer norm → a linear LM head producing the next-token logits → cross-entropy loss against targets (which are just the inputs offset by one). 55:50

The block: a residual pathway around attention (communication) and the MLP (computation). With block size 8 there are 8 nodes; node 1 attends to itself, node 2 to {1,2}, node 3 to {1,2,3}, and so on — "you apply layer norm, then self-attention so these 8 nodes communicate." The MLP is a 2-layer net with GELU acting per-node. 57:39

The attention code is "the meat": compute Q, K, V from X; matrix-multiply Q·K (the dot product, batched across all queries/keys/heads); masked-fill the forbidden (future) positions with -infinity so softmax drives them to zero ("otherwise the future gives away the answer"); softmax → attention matrix; multiply by V (the weighted sum); linear projection back. 59:30

Encoder vs decoder is one deleted line: "To get an encoder, just delete the mask line — all nodes communicate. For cross-attention, add a line where the queries come from X but keys/values come from the top of the encoder." Decoder-only = GPT; encoder-only = BERT (trained with masking/denoising instead of next-token); encoder-decoder = T5. 62:50

🏆 Why Transformers Won

The transformer simultaneously optimizes three desirable properties: 43:40

Property	Why it matters
1. Expressive	Forward pass can implement very interesting functions, potentially even meta-learning
2. Optimizable	Residual connections, layer norms — gradients flow easily, not too many hops from supervision to input
3. Efficient	A shallow-wide graph, perfect for GPU parallelism ("the most under-appreciated — in deep learning, scale matters")

RNNs vs transformers: "In principle RNNs can implement arbitrary programs — but that's a useless statement. They're not optimizable and not efficient, because they're serial computing devices. Stretch out an RNN and it's a long, thin compute graph — bad for backprop (too many steps). A transformer is shallow and wide: few hops from supervision to input, along residual pathways, all processed in parallel." 65:40

His renaming: "In hindsight I'd have called the paper 'a general-purpose, efficient, optimizable computer' instead of 'Attention Is All You Need.'" And the closing synthesis: "If you scale up the training set and use a powerful enough neural net like a transformer, the network becomes a kind of general-purpose computer over text. Instead of performing a single fixed sequence, you design the sequence in the prompt." 67:20

✅ Key Takeaways

Three programming paradigms, stacked. Software 1.0 = design the algorithm (70 years). Software 2.0 = design the dataset (the data engine compiles data into weights). Software 3.0 = design the prompt (condition an LLM). They layer, not replace.
An LLM is a next-word predictor — and that's the whole trick. All of its task ability comes from narrowing the prediction to the right "slice" (few-shot, step-by-step, IQ-200 conditioning).
Prompt wording matters measurably. "Let's think step by step" took the juggler problem from 17% → 78.7%; "let's work this out step-by-step to be sure we have the right answer" hit 82%.
The model is a simulator. Linux terminals, fake ping results, smart-home brains, GPT-as-backend, the Sydney persona — "you can condition it into arbitrary universes" via prompt alone.
The hottest new programming language is English — and prompting is how you program humans too. The tech is converging toward humans.
Attention = message passing on a directed graph. Queries look for things, keys say what you have, values are what you communicate. Encoder vs decoder is literally one deleted mask line.
Transformers won on three axes at once: expressive, optimizable (residuals + layer norms), and efficient (shallow-wide = GPU-friendly). "In deep learning, scale matters" — efficiency is the underrated one.
The 2017 architecture is still the architecture. The only durable change since has been pre-norm. "The GPTs you see today are basically the 2017 transformer."

🔗 Resources & Links

📺 Original video — the full lecture
💻 nanoGPT — the ~300-line GPT-2 reproduction

📍 Timestamp Index

0:00 Intro & background (OpenAI → Tesla → OpenAI)

2:32 Why it's interesting to hack now

3:09 Software 1.0 — instructions to a computer

4:48 Software 2.0 — the data engine

7:04 LLMs — next-word prediction magic

8:35 The juggler prompt — step-by-step

10:26 Conditioning the predictor — IQ-200

11:22 The model as simulator (terminal, ping, smarthome)

16:20 GPT-as-backend + Sydney prompt

18:45 Prompt engineering as a job (Riley Goodside)

20:26 Software 3.0 — English as the language

30:30 Pre-2012 — the feature-descriptor zoo

33:57 The brain convergence hypothesis

37:40 The origin of "attention" (Bahdanau email)

39:20 Attention Is All You Need — delete the RNN

41:17 In-context learning / meta-learning

44:25 Attention = message passing on graphs

52:10 nanoGPT walkthrough (tiny-Shakespeare)

57:39 The transformer block — communicate + compute

65:40 RNNs vs transformers — shallow-wide wins

67:20 General-purpose computer over text — conclusion