Yann LeCun: World Models: Enabling the next AI revolution

ELI5 / TLDR

One of the men who built modern AI got up in front of a room of researchers and said: the thing everyone is excited about is a dead end. Chatbots learn from text, and text is a tiny, tidy slice of reality. A four-year-old has already absorbed as much raw information through their eyes as a chatbot gets from all the writing on the internet. LeCun’s bet is that real intelligence comes from watching the world and quietly building a private mental model of how it behaves — what he calls a “world model” — and that the way to build one is not to make the machine paint detailed pictures of the future, but to make it predict the future in a stripped-down, abstract sketch where the unpredictable junk has been thrown away.

The Full Story

”Machine learning sucks”

That is genuinely how LeCun opens. The point underneath the provocation: machines are spectacularly bad at the things a small child finds trivial. A ten-year-old can clear a dinner table the first time you ask. A teenager learns to drive in about twenty hours. Self-driving car companies have millions of hours of driving footage and still can’t match that teenager’s reliability.

This is an old observation with a name — the Moravec paradox. The things that feel hard to us (chess, proving theorems, doing integrals) turn out to be easy for computers. The things that feel effortless (picking up a cup, knowing a dropped object falls) turn out to be brutally hard. LeCun’s framing: language is the easy part. The messy, continuous, high-dimensional real world is the hard part, and current AI doesn’t really touch it.

So what is intelligence, if not knowing a lot or having a lot of skills? LeCun, leaning on the developmental psychologist Jean Piaget, lands on: intelligence is what you do when you don’t know. It’s the ability to face a brand-new task and handle it without prior training. By that definition, a chatbot — which is essentially a vast, searchable pile of facts and learned skills — is impressive but not intelligent. And the term “AGI,” he says, is nonsense: human intelligence is specialized. None of us can do everything. What we can do is adapt fast.

The 400,000-year argument

Here’s the calculation that does the heavy lifting, and it’s worth going slowly.

A typical chatbot today is trained on roughly 20 trillion words. Stack up all that text and a human reading non-stop would need about 400,000 years to get through it. Sounds like an unbeatable amount of information.

Now look at a four-year-old child. Think of the eyes as a pair of data cables running into the brain — about two million nerve fibres, each carrying a trickle of information every second. Run that for the roughly 16,000 waking hours a four-year-old has lived, and the total amount of information that child has taken in through vision alone is about the same as those 400,000 years of text.

“Four-year-old world through vision, same amount of data as 400,000 years through text.”

A four-year-old, in four years, matches the entire written internet — just by looking around. LeCun’s conclusion is blunt: you are never going to get human-like intelligence by training on text. The real river of information is sensory, and it’s orders of magnitude wider than language.

Two ways a machine can “think”

Now the architectural heart of the talk. LeCun draws a line between two fundamentally different ways a system can produce an answer.

The first way — how chatbots work — is to shove the input through a fixed stack of layers and out pops an answer. Same amount of computation every time, no matter how hard the question. Picture a conveyor belt: the question goes in one end, gets processed by exactly the same machinery, and the answer drops out the other. To make a chatbot “reason,” you trick it into generating more words, because more words means more passes down the belt. But, LeCun says, that isn’t reasoning. We don’t think in words. We think in some internal space and only translate to words at the end.

The second way is to search for the answer. You imagine a possible answer, then ask: how well does this fit the situation? Adjust, ask again, repeat — until you’ve found the answer that fits best. LeCun calls the “how badly does this fit” meter an energy function: a good fit means low energy, a bad fit means high energy. Thinking, in this view, is rolling downhill to the lowest-energy answer. The big advantage is that you can spend more search effort on a harder problem. The conveyor belt can’t do that.

The world model, and the guardrails

Now bolt a world model onto that search. The recipe:

You perceive your surroundings and form a sense of the current state.
You imagine a sequence of actions.
The world model — your internal simulator of how things behave — predicts what would happen if you took those actions.
An objective checks: did that get me what I wanted?
You search for the action sequence that scores best.

If “world model” and “imagine actions and check the outcome” sound like a planner, that’s exactly right — engineers have done versions of this since the 1960s under the name model predictive control. The novelty is using a learned world model rather than hand-written equations.

A nice side effect: safety can be baked in. Alongside the task objective (“open the door”), you add guardrail objectives — never take the world through a state that hurts someone. The system literally cannot produce an action that violates them, because it’s optimizing against them at all times. Contrast a chatbot, which you can only make safe by fine-tuning after the fact — and which someone can always jailbreak. A planner that minimizes a guardrail objective has nothing to jailbreak.

And ultimately we want this to be hierarchical. To get from his NYU office to Paris tomorrow, LeCun doesn’t plan 10-millisecond muscle twitches — he can’t, and he doesn’t have the information (how long until a taxi stops?). He plans coarsely at the top — get to the airport, catch a plane — then refines each step into sub-goals. Getting machines to do this layered planning is, he says cheerfully, completely unsolved. A great PhD topic.

The trap LeCun spent ten years stuck in

So how do you build a world model? The obvious instinct is a generative model — train it to predict the next video frame the way a chatbot predicts the next word. LeCun tried exactly this for about ten years and mostly failed, and he now thinks the failure was inevitable.

The reason is beautiful in its simplicity. With text, the next word is one of a finite list, so the machine can hedge across all of them. With video, the future is effectively infinite. Imagine filming a lecture hall, slowly panning the camera, then stopping and asking the machine to continue the video. It can guess there’s a room and maybe windows. It has no way to predict what every face looks like or which chairs are empty — that information simply isn’t in the input. Force it to predict the unpredictable and you “kill it”: it gives up and predicts a blurry average of everything that could happen.

“When you train a system to make this kind of prediction, you kill it.”

(And the AI video generators making cute clips? They don’t refute this — they only need to produce one plausible video, not represent all of them. A much easier job, and not the same as understanding the world.)

JEPA: predict the gist, not the pixels

LeCun’s fix, and the centerpiece of his research program, is JEPA — Joint Embedding Predictive Architecture.

The trick: stop predicting raw pixels. Instead, pass both the “before” and the “after” through an encoder that boils each down to an abstract summary — a representation — and predict the summary. Why does this rescue the situation? Because the encoder is free to throw away everything that’s unpredictable. It keeps the gist (a ball is flying through the air) and discards the noise (the exact pattern on the wall behind it). Prediction in this stripped-down space is coarser but far more accurate.

This is exactly what science does, and LeCun makes the analogy explicit. To model airflow over a wing, you don’t track every air molecule — you’d never finish, and the tiny errors would explode. You work at the abstract level of velocity and density and get useful predictions. Every level of science — particles, atoms, molecules, cells, organisms, societies — is an abstraction that ignores the messy level below so it can predict further ahead. A world model should be the same: an abstraction, not a pixel-perfect simulator. Which is why, he insists, the fashionable phrases miss the point — a world model is not a “digital twin,” not a generative model, not a video generator.

The one thing that can go wrong: collapse

JEPA has a built-in failure mode, and it’s almost funny. If your only instruction is “make the before-summary and after-summary match,” the system finds a cheat: output the same blank summary for everything. Now they always match perfectly. Prediction error: zero. Usefulness: also zero. This is collapse — the system stops paying attention to its input entirely.

Most of the technical craft of JEPA is about preventing collapse, and LeCun’s preferred approach is information maximization: don’t just demand that summaries match, also demand that they stay rich and varied. Make the machine use every dimension of its representation to carry genuinely different information, so it can’t shrink everything to a constant.

His newest tool here, which he expects to push hard in coming years, is SIGReG (sketched isotropic Gaussian regularization). The idea, kept at the level of intuition: take a batch of summaries, scattered as points in a high-dimensional space, and gently nudge them until they spread out into a nice symmetric, well-balanced cloud — a shape where every dimension is doing independent work. You can’t measure that shape directly in 2,000 dimensions, so the clever bit is to repeatedly shine a flashlight through the cloud from many random angles, fix the shadow along each direction, and a theorem promises that if you do this from enough angles, the whole cloud ends up balanced. It runs on a single GPU; the open problem is scaling it.

A parallel family of methods, called distillation (the techniques behind models named DINO and V-JEPA), is what’s actually been scaled up so far and gives the best image and video representations available today.

It actually learns common sense

The payoff, and the part LeCun seems most pleased by. Train V-JEPA to predict masked-out chunks of video, then watch its internal prediction error as it views new clips. Show it something physically impossible — a thrown ball that vanishes in mid-air — and the prediction error spikes. The machine is surprised. It expected the ball to keep going and it didn’t.

This is exactly how psychologists test whether a human baby has learned a concept: show a violation (a car rolling off a ledge and floating instead of falling) and see if the baby stares longer. A six-month-old shrugs; a ten-month-old is shocked. V-JEPA does the ten-month-old thing. As far as LeCun knows, it’s the first time a fully self-supervised system — one that learned purely by watching, with nobody labelling anything — picked up a working sense of what is and isn’t physically possible. And when they bolted a small add-on onto V-JEPA to judge distance, it could estimate depth from a single image, beating a strong competitor — meaning the system had figured out, just from filling in blanks in video, that the world is three-dimensional.

The closing heresy

LeCun ends with a list of things to abandon: generative models, probabilistic models, and (mostly) reinforcement learning — which he calls “what you do when you’re desperate,” so sample-hungry it should be the last resort, not the first. His advice to academics is even sharper: don’t work on chatbots at all — there’s nothing you can add that the big labs aren’t already doing with more compute.

“This does not make me very popular in Silicon Valley.”

He notes, almost in passing, that he left Meta at the end of last year and started a company, AMI Labs, aimed squarely at “AI for the real world” — robots, industrial processes, anything high-dimensional and noisy where chatbots are, in his word, helpless.

Key Takeaways

The Moravec paradox is the core obstacle. What’s easy for humans (physical common sense) is hard for machines; what’s hard for humans (chess, math) is easy. Current AI is stuck on the easy-for-humans side.
Intelligence is adaptation, not accumulation. Not how much you know or how many skills you have, but how fast you handle something genuinely new. By this measure “AGI” is a category error — human intelligence is specialized.
Vision dwarfs text as a data source. A four-year-old absorbs as much through their eyes in four years as a chatbot gets from all internet text (~10^14 bytes, the “400,000 years” of reading). Text-only training can’t reach human-like intelligence.
Two modes of inference. Chatbots use a fixed forward pass (same compute every time). LeCun wants inference-by-search: imagine an answer, score it with an energy function, optimize. Lets you spend more effort on harder problems.
World model + planning + guardrails. Imagine actions, predict outcomes, pick the best — classic model predictive control, but with a learned model. Guardrail objectives make the system intrinsically safe and un-jailbreakable.
Don’t predict pixels. The future of video is effectively infinite; forcing pixel prediction produces blurry averages and “kills” the model. Predict in abstract representation space instead — keep the gist, discard the unpredictable.
JEPA’s danger is collapse — the system outputting a constant blank summary that trivially “matches.” Preventing collapse (via information maximization / SIGReG) is most of the engineering.
V-JEPA learned physical common sense unsupervised. Prediction error spikes at impossible events (a vanishing ball), mirroring the “violation of expectation” test used on human babies. It also recovered 3D depth from single images.
LeCun’s hit-list: abandon generative models, probabilistic models, and (mostly) reinforcement learning; if you’re in academia, don’t work on LLMs at all.

Claude’s Take

This is LeCun doing what LeCun does: planting a flag far from the crowd and daring you to bet against him. The talk is a tight, honest distillation of a research program he’s pushed for years (the 2022 position paper, JEPA, energy-based models), and the intellectual through-line — intelligence needs grounding, grounding needs a world model, and a world model must be abstract not generative — is genuinely coherent. The “400,000 years vs. four-year-old” calculation is the kind of back-of-envelope argument that reframes a whole debate, and it lands. The V-JEPA surprise-at-impossible-events result is the strongest evidence on offer, and it’s a real result, not a vibe.

Where to keep your skepticism. First, LeCun has a track record of confidently declaring approaches dead that then kept delivering — he’s been writing off the LLM paradigm for years while it kept getting more capable. “Text-only can’t reach human intelligence” may well be true, yet the frontier keeps moving in ways that complicate the bet. Second, the demos here are honest but small: planning in simulated toy environments (a double pendulum, a block-pushing task), with “scale it up” doing a lot of load-bearing work — exactly the step that has historically humbled elegant alternatives to the dominant paradigm. Hierarchical planning, which he admits is the thing you’d actually need, remains completely unsolved by his own account. Third, there’s a man-with-a-new-company incentive in the air: this talk doubles as the founding thesis of AMI Labs, so the confidence is partly a pitch.

None of that makes him wrong. The critique of pixel-prediction is airtight, the energy-based framing is intellectually serious, and “vision is the real firehose” is hard to argue with. This is the clearest single hour you can spend understanding why a top-tier researcher thinks the current boom is built on sand — even if the alternative is still mostly promise. A 9 because it’s rigorous, original, honestly hedged, and explains a contrarian worldview better than most papers do; not a 10 because the empirical evidence for the alternative is still thin and the bet remains unproven.