Jitendra Malik: Vision and Robotics for Embodied AI

ELI5 / TLDR

A robot has three jobs: walk around without falling (locomotion), find its way to things (navigation), and use its hands to do stuff (manipulation). Malik says the first two are basically solved. The third one — picking up an egg and cracking it without making a mess, every single time — is where everyone is stuck, and it’s stuck because there’s almost no good training data for hands. His proposed fix: stop paying people to puppeteer robots, and instead let robots learn by watching ordinary YouTube videos of humans doing things, then let them practice in their own bodies, the same way a toddler learns by watching a parent wash dishes.

The Full Story

Malik opens by carving robotics into three problems: navigation, locomotion, and manipulation. He then delivers his verdict early and bluntly.

“Locomotion, we have good progress. Navigation is nearly done. But I will argue that in manipulation, we have a long way to go.”

The two problems that are mostly solved

For navigation, he shows a system called “go to anything” that passes what he calls the Airbnb test. Drop a robot into a house it has never seen, give it a goal — “find the television,” “find the red cup on the brown leather chair” — and it explores until it finds it. As it goes, it builds a partial map. Not a complete blueprint, just a semantic map: a memory of where objects are. Ask for the same thing again and it goes straight there, no re-searching. This is how you operate too — you don’t re-survey your kitchen every time you want a glass.

One quiet but important finding: the system that worked was modular, not end-to-end. Meaning, instead of training one giant network to go from camera pixels all the way to motor commands, you plug in the best available vision model for seeing, and only use learning for the part that genuinely needs it — figuring out efficient places to explore. End-to-end “did not do very well.”

For locomotion — walking — he shows quadrupeds and humanoids hiking around Berkeley and up San Francisco’s steepest hills (a 41% slope). The striking part: the robot was trained entirely in simulation and then dropped into the real world cold, with no real-world tuning. The jargon for this is “zero-shot,” which just means it had never practiced on the actual hill and walked up it anyway.

Why hands are so much harder

The bottleneck for manipulation is data, and the reason cuts deep. Language models swim in text — Malik estimates a large language model trains on roughly 10,000 times more words than a human child ever hears. For hands, the useful data isn’t text or even video. It’s sensory-motor trajectories: the moment-by-moment record of what your body did and what it felt while doing it. We don’t have those recordings even for humans, let alone robots. There is no YouTube of fingertips.

So you can’t be wasteful the way language research can. You have to think carefully about the structure of the problem. Malik offers a three-layer hierarchy, and later gives each layer a patron saint:

The level of Aristotle — goals and plans. “I want to make an omelet.” Language models are genuinely good here; ask one and it tells you to crack eggs into a bowl.
The level of Euclid — the actual motion path your hand traces through space and time. Language is useless here. The phrase “pick up” is the same whether you’re lifting a sugar cube or a heavy tray, but the real movements are completely different. Language compresses; it throws away exactly the detail you need.
The level of Newton — forces. To change anything in the world you have to push on it. Picking up a glass means gripping hard enough that friction stops it slipping, while pulling up against gravity, while tiny sensors in your fingertips report back. Pure vision can’t see any of this.

His summary equation: action = movement + goal. Language handles the goal. It cannot handle the movement.

He also plants a flag he refuses to defend, calling it his credo: real physical intelligence needs multi-fingered hands, not the two-pronged pincers (“parallel jaw grippers”) most labs use. If that $250 bottle of wine matters to you, you want a full grip, not a two-point pinch.

Why the three obvious solutions all disappoint

How do you get the missing data? There are three standard answers, and Malik knocks each one down.

Teleoperation — a human wears a rig and puppeteers the robot; the robot’s movements get recorded. This is the dominant approach, with “thousands or millions of people all around the world doing this really painful job.” Why painful? Because it forces a human to behave like a robot, working off vision alone with no sense of touch.

“It’s really it’s like a torture session almost… those trajectories are recorded like this, they are very slow.”

That’s why the polished demos you see are quietly sped up 5x. And the slowness isn’t fixable with money — it’s a hard limit of a human operating blind to touch.

Human videos — watching egocentric (head-cam) footage of people doing tasks. As a vision person he loves this, and YouTube is essentially infinite. The catch: a human body is not a robot body (the “embodiment gap”), and video still shows you nothing about contact forces.

Simulation — train inside a physics engine, where forces come for free. His tell: simulation is the one method that actually worked for navigation and locomotion, so it should be your default prior. The real objection isn’t the “sim-to-real gap” (he waves that off as mere engineering). It’s that defining the reward — the score that tells the robot it did well — is brutal for manipulation. A pouring task might need 150 hand-tuned terms with coefficients, and people simply give up and fall back to copying teleoperation demos.

The toddler, and the way out

Having dismantled all three, Malik reaches for Alan Turing’s 1950 suggestion: don’t try to build an adult mind, build a child’s and then educate it. His one chosen nugget is what imitation learning actually is.

The story you’re told is that imitation = “behavior cloning,” as if the adult reaches into the child’s brain and drives her muscles. That’s not how it works. A girl watches her father wash dishes. She cannot copy him directly — she has tiny hands (embodiment gap) and can’t feel the forces he feels (missing data). So she extracts a hint from watching, then learns the task in her own body through play and trial and error.

“Just replace the girl by a robot. Done. I intend to devote the next few years of my life to making this slide as the way to solve robotic manipulation.”

He shows this already working — but for locomotion, the easier case. They filmed ordinary RGB video of humans moving, did 3D reconstruction, retargeted the motion onto a humanoid in a simulator, then used reinforcement learning to make it robust. Only ~100 trajectories were needed, not thousands of hours, because the simulator generates the variety. This recipe is now widely deployed for humanoids.

The computer-vision plumbing that had to be invented first

Extending this to hands meant solving a hard vision problem: turning flat RGB video into a moving 3D scene (“4D-fy it”) so you know where everything is and roughly what’s touching what. Malik’s lab spent years 3D-reconstructing humans, then hands, then — the real surprise — objects, which turned out to be the hardest part.

The trick is treating 3D not as classic multi-view geometry (where you need the camera to move enough to triangulate) but as reconstruction by recognition. You see an object once, it connects to a memory of similar shapes you’ve seen from all angles, and you retrieve a full 3D model. Their system (Sham3D, appearing at CVPR) reconstructs full 3D objects from a single image, even objects not in its training set — because once you’ve modeled a million objects, a new one is just a remix of old ones. Why only now, after 150 years of photogrammetry? Data. Getting paired image-to-3D training data for cluttered real scenes is hard; they bootstrapped it with CAD models, then human judges picking the better of two reconstructions, then graphics artists for the worst cases.

Forces, the part nobody wants to do

Finally he returns to touch. He shows a fancy tactile sensor (Digit 360) and admits it’s probably overkill, then makes his real point with a bar chart: vision alone gets you decent performance, but vision + proprioception (sense of joint position) + touch nearly matches ground truth.

“You will not get to 99, you will get to 85. And that may be all the difference between being real or not.”

And better sensing means fewer demos — a few hundred instead of a few thousand. He closes with a peeling-vegetables task that needs both position and force control, trained using a human judge to rate good peels from bad (borrowing the “AI feedback” trick from language models). Michelin restaurants are safe for now.

His parting shot at the field: obsessing over vision-language-action models is “a distraction.” Translating “omelet” into a recipe is the easy part. Cracking the egg 99.99% of the time, generalizing across objects and clutter — that’s the level of Euclid and Newton, “the good old fight of contact forces, trajectories.”

Key Takeaways

Navigation and locomotion are essentially solved; dexterous manipulation is the open frontier — and the gap is entirely about data, not algorithms.
The useful data for hands is sensory-motor trajectories (what the body did + what it felt), which barely exist even for humans. This forces frugality that language research never needed.
Three levels of motor control: goals/plans (language is good here), motion paths in 4D (language is useless), and forces (vision is blind to these). Action = movement + goal.
Modular beats end-to-end for navigation: use a strong pretrained vision model, reserve learning for the part that needs it (exploration). End-to-end overfits to the few trajectories it saw.
Teleoperation is a dead end for quality — it makes humans act like robots with no touch feedback, producing slow data that gets secretly sped up 5x for demos. Not fixable with money.
The toddler model: learn affordances by watching video, then master the task in your own body via trial and error. Embodiment gap and missing force data exist for a watching child too, not just robots.
Simulation is the proven path (it cracked navigation and locomotion). The hard part isn’t sim-to-real realism (just engineering) — it’s hand-designing reward functions for manipulation, which can balloon to 150 tuned terms.
“Reconstruction by recognition”: reconstruct 3D from a single image via learned memory of past shapes, not camera parallax. Once you’ve modeled a million objects, new ones are remixes.
Touch is non-negotiable — vision + proprioception + touch nearly matches ground truth, and richer sensing cuts the number of demos needed by ~10x.
You don’t need detailed physics: the world reaches the robot through a low-dimensional interface (its joints and contact points), so even crude simulators can be “good enough” — like how three cone types capture infinite light wavelengths.

Claude’s Take

This is a senior figure (Malik essentially helped found modern computer vision) giving a clear-eyed state-of-the-field talk, and the value is in the framing more than any single result. The three-level hierarchy — Aristotle, Euclid, Newton — is a genuinely useful mental scaffold, and his demolition of teleoperation lands because it names something the field half-hides: those slick robot demos are sped up because the data collection is miserable and slow. That’s an honest thing to say out loud.

Where to keep your guard up: he is selling his own research program, so the framing is tilted toward it. “Navigation is nearly done” and “we’ve solved 3D hands” are the confident statements of a man whose lab works on exactly those things; other researchers would contest both. His central bet — that learning from raw YouTube video plus in-body practice will crack manipulation — is, by his own admission, six years in and “close to solved, but not fully solved,” which in research-speak can mean anything. The demos are cherry-picked (he says so himself, repeatedly and to his credit). And dismissing the sim-to-real gap as “just engineering” is the kind of line that’s easy to say and famously hard to cash; the audience pushed back on exactly this and his answer (“all models are wrong but some are useful”) is true but evasive.

Score is an 8: high signal, intellectually honest about its own limits, and the conceptual frame outlasts the specific results. Not a 9 because there’s little here a careful follower of the field hasn’t heard, and the strongest claims are asserted rather than proven.