What Happens After A 1,000,000x AI Compute Leap?

ELI5/TLDR

Jeff Dean runs the science side of Google’s AI work, and here he talks shop with an enthusiastic interviewer. The big claims: we are not running out of data to train AI (there’s video, there’s machine-made data, and clever tricks to squeeze more from what we have); the bottleneck has quietly shifted from building models to using them, which changes what kind of computer chips matter; and if computing power gets a million times cheaper again over the next decade — as it did over the last — we might hand a machine a one-line request and have it design an airplane in five days. Also, cosmic rays really do corrupt computer memory, and Google has the data to prove it.

The Full Story

The data isn’t running out

There’s a common worry that AI has read the whole internet and is about to run out of things to learn from. Dean doesn’t buy it.

His first point: text is only one kind of data. Imagine you’ve read every book in a library but never watched a film. There’s an enormous amount of video on the internet that today’s models barely touch. That’s a fresh reservoir.

His second point is subtler, and it’s worth slowing down on. You can make AI generate its own training material — and then learn from it. This sounds like a snake eating its own tail, and the interviewer pushes back: when you train an AI on stuff a different AI made, doesn’t everything collapse into mush?

Dean’s answer hinges on a technique where the machine tries things and checks whether they worked. Picture teaching a model to write code. You ask it to solve a problem, and instead of one attempt, it fires off a thousand different attempts. Then you run them through a sieve. Doesn’t even compile? Bin it — that’s 800 gone instantly. Doesn’t pass the tests? Gone. What survives is a small pile of solutions that demonstrably work, and the best of those becomes new, high-quality teaching material. The machine generated its own lesson, but reality — does the code run? — did the grading. (The technical name for “try a lot, keep what scores well” is reinforcement learning; the thousand attempts are “rollouts.”)

Then comes a trick he’s clearly fond of. Once you have a program that works, you can ask the model to rewrite it in a different programming language — Python into Go, say — and now you’ve got brand-new Go training data, fully correct, for free. He notes this is a far richer move than the old way of multiplying data. Years ago, to give an image-recognizer more to chew on, you’d just nudge a photo a few pixels sideways and call it a new example. Translating a whole working program into another language is augmentation on a completely different level.

“Your prompt is the fully specified behavior of the system you want and you just want it in a different language.”

The quiet shift from training to inference

Here’s a fact that surprised even the interviewer: in modern data centers, something like 90% of the AI computing work is no longer training — it’s inference. Two words worth pinning down. Training is building the model, the long expensive education. Inference is the model actually doing its job once it’s built — answering your question, running an autonomous agent in the background. Building the brain is now the small part; using the brain is the flood.

Why does that matter? Because the two jobs have different appetites, and chips can be tuned for one or the other. When a model is just answering questions, the weights inside it don’t change, you’re handling a huge volume of requests, and — crucially — you can get away with sloppier math.

That “sloppier math” point is genuinely strange, and both men dwell on it. Numbers inside these models are normally stored with lots of decimal precision. It turns out you can crush them down to almost nothing — a format called FP4, four bits, which can only represent a handful of distinct values — and the model still works.

“If you told that to a computer scientist from 15 years ago they’d be like… that’s not enough numbers.”

Think of it like compressing a photo so hard it should turn to static, and somehow the picture survives. The reason it works involves clever reshuffling of the numbers beforehand, plus a small shared “scaling factor” sprinkled in every 64 or 128 values to claw back a bit of accuracy. Dean isn’t sure how much lower they can push — people are poking at two-bit and one-bit numbers — but the direction is clear: less precision, less energy, more specialization. Google’s newer TPU chips (their in-house AI processors) are built around exactly this.

Stop separating learning from doing

Today, training a model and then fine-tuning it afterward are two distinct stages — first one, then the other. Dean finds this “intellectually dissatisfying.” The natural thing, he argues, is to interleave them: learn a bit, go act in the world, see what happened, learn from that, repeat.

His reasoning is that you learn far more from doing than from watching. Write code and see if it runs, and you’ve genuinely learned something. Just sitting there “seeing tokens stream by you” — which is what ordinary training is — teaches less. Do enough small learn-act cycles, he notes, and “a summation starts to look more like an integral” — the steps blur into something continuous.

The catch the interviewer raises: if a model never stops learning, how do you ever certify it’s safe to release? Dean’s fix is sensible — let it learn continuously behind the scenes, but freeze a version, run the safety checks and red-teaming on that frozen snapshot, ship it, and keep the learning going underneath for the next release.

The million-X question

The headline. Nvidia’s Jensen Huang likes to say computing power got a million times more capable over the last ten years. So: another million-fold in the next ten — what becomes possible?

Dean reaches back to set the scale. Ten years ago, AI language models were primitive — the basic ideas that led to today’s systems had only just appeared, and the models of that era now look “ancient.” Project that same leap forward and he refuses to see the pace slowing. His concrete dream: engineering and science tasks that today take large teams years — designing an airplane, designing a new computer chip — collapsing into days, driven by swarms of cooperating AI agents that break a big job into small ones. He’s careful to say we’re not there yet.

Big models exist to teach small ones

A nice reframing of the open-versus-closed-model debate. Google’s small, fast “Flash” models are good because they’re taught by the big expensive “Pro” models — a process called distillation, where a large model’s knowledge is poured into a smaller one. The implication: you have to keep building enormous, inefficient frontier models, because that’s the only way to get cheap, fast, nearly-as-good small ones. The small models are “the workhorse of what people generally want to use.” There is, he admits with a smile, also some “magic sauce” they won’t reveal.

Faking infinite memory, and cosmic rays

Two more threads. First, the dream of a model that holds everything — your whole life, or all 10 billion lines of Google’s code — in mind at once. The obstacle is that the standard attention mechanism gets quadratically more expensive as the input grows (double the text, quadruple the cost — written O(n²)). Dean’s vision is a cascade that fakes total recall: a cheap retrieval pass narrows 10 billion documents to 30,000, a light model narrows those to 117, and only those land in the expensive context window. You’d get the illusion of having the whole internet at your fingertips without paying for it.

Second, the failure stories. Google once ran an internal chat group called “data centers on fire” — sometimes literally, when a bus bar overheated. The principle from day one was building “reliable systems out of unreliable parts” — early Google ran on cheap consumer machines with no error-correcting memory, so they checksummed data by hand. And yes, the cosmic-ray thing is real: stray particles flip bits in memory, and Google can see it in the data — clusters of machines facing one direction on Earth show a brief spike in memory errors that the machines on the far side don’t.

Key Takeaways

Inference now dominates compute. ~90% of ML work in data centers is inference (using models), not training (building them). This justifies specialized inference chips — Google’s TPU “8i”/“8t” — tuned for low precision, high request volume, fixed weights.
FP4 works. Four-bit number formats are viable for inference when paired with distance-preserving transforms and a shared scaling factor every ~64–128 weights. Two-bit and one-bit integer formats are being explored.
Synthetic data is fine if reality grades it. RL rollouts generate many candidate solutions; cheap filters (compiles? passes tests?) discard most, leaving high-quality examples to fold back into training.
Cross-language translation is powerful data augmentation. A working program rewritten Python→Go yields correct new training data — far richer than pixel-shift augmentation.
Distillation is the strategy, not a side effect. Frontier models exist partly to teach smaller “Flash”-tier models, which are the real workhorses. Closed-vs-open is secondary to big-teaches-small.
Continual learning is the unsolved problem Dean has tried repeatedly and can’t crack. The aspiration: interleave passive learning with action-taking instead of separate train-then-tune phases.
Long context via cascade, not brute force. Attention is O(n²); the plan is layered retrieval (10B docs → 30K → 117 → context window) to fake near-infinite memory cheaply. ~100 papers exist on sub-quadratic attention.
Cosmic rays flip real bits. Confirmed in Google’s ECC monitoring — directional clusters show transient single-bit error spikes. Single consumer machines (parity only, no ECC) are mostly fine; fleets of tens of thousands are not.
The million-X bet: another decade of ~1,000,000x compute could compress multi-year engineering (aircraft, chip design) into days via multi-agent workflows. Not there yet.

Claude’s Take

The format is an enthusiast interviewing a heavyweight, and you have to do some separating. The host’s “I couldn’t believe it,” “crazy,” “unbelievable” register is texture, not evidence — strip it out. What’s left underneath is unusually solid, because Jeff Dean is about as credible as sources get on this material: he co-built the systems (MapReduce, TensorFlow) that the modern field runs on, and he’s not selling anything in a casual chat.

The genuinely useful claims are the unsexy structural ones. The training-to-inference shift (90%) is a real and underappreciated fact that reshapes the entire hardware economy — it’s why every chipmaker is suddenly talking about inference. FP4 working is the kind of thing that shouldn’t be true and is, and Dean’s matter-of-fact treatment of it is more convincing than any hype. The distillation reframing — frontier models as expensive teachers whose whole point is producing cheap students — is a clean mental model that cuts through a lot of confused commentary.

Where to keep your guard up: the million-X-equals-airplane-in-five-days bit is the one moment Dean lets himself dream, and he flags it himself (“we’re not there yet”). Treat it as aspiration, not forecast. Note too that he’s an interested party — he runs Google’s AI science and naturally frames things favorably (the “magic sauce” dodge is honest about being a dodge). And the “we’re not running out of data” line is genuinely contested in the field; his arguments are good but not the last word.

Score: 8. High signal-per-minute from a top-tier source, several mental models worth keeping, and refreshingly little salesmanship. Docked from higher only because it’s a grab-bag of topics rather than a deep dive on any one, and the lightning round, while charming, is filler.

What Happens After A 1,000,000x AI Compute Leap? | Jeff Dean