Leopold Aschenbrenner 2027 Agi China Us Super Intelligence Race And The Return Of History
read summary →TITLE: zdbVtZIn9IM CHANNEL: Unknown DATE: ---TRANSCRIPT--- What will be at stake will not just be cool products But whether liberal democracy survives, Whether the CCP survives, what the world order for the next century is going to be The CCP is going to have an all out effort to infiltrate American AI labs Billions of dollars, thousands of people CCP is going to try to out-build us. People don’t realize like how intense state level espionage can be When we have like literal superintelligence They can like Stuxnet the chinese data centers You really think that will be like a private company And the government wouldn’t be like “oh my god what is going on?” I do think it is incredibly important that these clusters are in the united states I mean would you do the manhattan project in the UAE? 2023 was the moment for me when it went from AGI as this sort of theoretical, abstract thing, and you’d make the models to like, I see it, I feel it. I can see the cluster where it’s trained on, like the rough combination of algorithms, the people, like how it’s happening, and I think most of the world is not; most of the people who feel it are like right here Today I’m chatting with my friend Leopold Aschenbrenner. He grew up in Germany and graduated as valedictorian of Columbia when he was 19. After that, he had a very interesting gap year which we’ll talk about. Then, he was on the OpenAI superalignment team, may it rest in peace. Now, with some anchor investments — from Patrick and John Collison, Daniel Gross, and Nat Friedman — he is launching an investment firm. Leopold, you’re off to a slow start but life is long. I wouldn’t worry about it too much. You’ll make up for it in due time. Thanks for coming on the podcast. Thank you. I first discovered your podcast when your best episode had a couple of hundred views. It’s been amazing to follow your trajectory. It’s a delight to be on. In the Sholto and Trenton episode, I mentioned that a lot of the things I’ve learned about AI I’ve learned from talking with them. The third, and probably most significant, part of this triumvirate has been you. We’ll get all the stuff on the record now. Here’s the first thing I want to get on the record. Tell me about the trillion-dollar cluster. I should mention this for the context of the podcast. Today you’re releasing a series called Situational Awareness. We’re going to get into it. First question about that is, tell me about the trillion-dollar cluster. Unlike most things that have recently come out of Silicon Valley, AI is an industrial process. The next model doesn’t just require some code. It’s building a giant new cluster. It’s building giant new power plants. Pretty soon, it’s going to involve building giant new fabs. Since ChatGPT, this extraordinary techno-capital acceleration has been set into motion. Exactly a year ago today, Nvidia had their first blockbuster earnings call. It went up 25% after hours and everyone was like, “oh my God, AI is a thing.” Within a year, Nvidia data center revenue has gone from a few billion a quarter to $25 billion a quarter and continues to go up. Big Tech capex is skyrocketing. It’s funny. There’s this crazy scramble going on, but in some sense it’s just the continuation of straight lines on a graph. There’s this long-run trend of almost a decade of training compute for the largest AI systems growing by about half an order of magnitude, 0.5 OOMs a year. Just play that forward. GPT-4 was reported to have finished pre-training in 2022. On SemiAnalysis, it was rumored to have a cluster size of about 25,000 A100s. That’s roughly a $500 million cluster. Very roughly, it’s 10 megawatts. Just play that forward half a year. By 2024, that’s a cluster that’s 100 MW and 100,000 H100 equivalents with costs in the billions. Play it forward two more years. By 2026, that’s a gigawatt, the size of a large nuclear reactor. That’s like the power of the Hoover Dam. That costs tens of billions of dollars and requires a million H100 equivalents. By 2028, that’s a cluster that’s ten GW. That’s more power than most US states. That’s 10 million H100 equivalents, costing hundreds of billions of dollars. By 2030, you get the trillion-dollar cluster using 100 gigawatts, over 20% of US electricity production. That’s 100 million H100 equivalents. That’s just the training cluster. There are more inference GPUs as well. Once there are products, most of them will be inference GPUs. US power production has barely grown for decades. Now we’re really in for a ride. When I had Zuck on the podcast, he was claiming not a plateau per se, but that AI progress would be bottlenecked by this constraint on energy. Specifically, he was like, “oh, gigawatt data centers, are we going to build another Three Gorges Dam or something?” According to public reports, there are companies planning things on the scale of a 1 GW data center. With a 10 GW data center, who’s going to be able to build that? A 100 GW center is like a state project. Are you going to pump that into one physical data center? How is it going to be possible? What is Zuck missing? Six months ago, 10 GW was the talk of the town. Now, people have moved on. 10 GW is happening. There’s The Information report on OpenAI and Microsoft planning a $100 billion cluster. Is that 1 GW? Or is that 10 GW? I don’t know but if you try to map out how expensive the 10 GW cluster would be, that’s a couple of hundred billion. It’s sort of on that scale and they’re planning it. It’s not just my crazy take. AMD forecasted a $400 billion AI accelerator market by 2027. AI accelerators are only part of the expenditures. We’re very much on track for a $1 trillion of total AI investment by 2027. The $1 trillion cluster will take a bit more acceleration. We saw how much ChatGPT unleashed. Every generation, the models are going to be crazy and shift the Overton window. Then the revenue comes in. These are forward-looking investments. The question is, do they pay off? Let’s estimate the GPT-4 cluster at around $500 million. There’s a common mistake people make, saying it was $100 million for GPT-4. That’s just the rental price. If you’re building the biggest cluster, you have to build and pay for the whole cluster. You can’t just rent it for three months. Can’t you? Once you’re trying to get into the hundreds of billions, you have to get to like $100 billion a year in revenue. This is where it gets really interesting for the big tech companies because their revenues are on the order of hundreds of billions. $10 billion is fine. It’ll pay off the 2024 size training cluster. It’ll really be gangbusters with Big Tech when it costs $100 billion a year. The question is how feasible is $100 billion a year from AI revenue? It’s a lot more than right now. If you believe in the trajectory of AI systems as I do, it’s not that crazy. There are like 300 million Microsoft Office subscribers. They have Copilot now. I don’t know what they’re selling it for. Suppose you sold some AI add-on for $100/month to a third of Microsoft Office subscribers. That’d be $100 billion right there. $100/month is a lot. That’s a lot for a third of Office subscribers. For the average knowledge worker, it’s a few hours of productivity a month. You have to be expecting pretty lame AI progress to not hit a few hours of productivity a month. Sure, let’s assume all this. What happens in the next few years? What can the AI trained on the 1 GW data center do? What about the one on the 10 GW data center? Just map out the next few years of AI progress for me. The 10 GW range is my best guess for when you get true AGI. Compute is actually overrated. We’ll talk about that. By 2025-2026, we’re going to get models that are basically smarter than most college graduates. A lot of the economic usefulness depends on unhobbling. The models are smart but limited. There are chatbots and then there are things like being able to use a computer and doing agentic long-horizon tasks. By 2027-2028, it’ll get as smart as the smartest experts. The unhobbling trajectory points to it becoming much more like an agent than a chatbot. It’ll almost be like a drop-in remote worker. This is the question around the economic returns. Intermediate AI systems could be really useful, but it takes a lot of schlep to integrate them. There’s a lot you could do with GPT-4 or GPT-4.5 in a business use case, but you really have to change your workflows to make them useful. It’s a very Tyler Cowen-esque take. It just takes a long time to diffuse. We’re in SF and so we miss that. But in some sense, the way these systems want to be integrated is where you get this kind of sonic boom. Intermediate systems could have done it, but it would have taken schlep. Before you do the schlep to integrate them, you’ll get much more powerful systems that are unhobbled. They’re agents, drop-in remote workers. You’re interacting with them like coworkers. You can do Zoom calls and Slack with them. You can ask them to do a project and they go off and write a first draft, get feedback, run tests on their code, and come back. Then you can tell them more things. That’ll be much easier to integrate. You might need a bit of overkill to make the transition easy and harvest the gains. What do you mean by overkill? Overkill on model capabilities? Yeah, the intermediate models could do it but it would take a lot of schlep. The drop-in remote worker AGI can automate cognitive tasks. The intermediate models would have made the software engineer more productive. But will the software engineer adopt it? With the 2027 model, you just don’t need the software engineer. You can interact with it like a software engineer, and it’ll do the work of a software engineer. The last episode I did was with John Schulman. I was asking about this. We have these models that have come out in the last year and none seem to have significantly surpassed GPT-4, certainly not in an agentic way where they interact with you as a coworker. They’ll brag about a few extra points on MMLU. Even with GPT-4o, it’s cool they can talk like Scarlett Johansson (I guess not anymore) but it’s not like a coworker. It makes sense why they’d be good at answering questions. They have data on how to complete Wikipedia text. Where is the equivalent training data to understand a Zoom call? Referring back to your point about a Slack conversation, how can it use context to figure out the cohesive project you’re working on? Where is that training data coming from? A key question for AI progress in the next few years is how hard it is to unlock the test time compute overhang. Right now, GPT-4 can do a few hundred tokens with chain-of-thought. That’s already a huge improvement. Before, answering a math question was just shotgun. If you tried to answer a math question by saying the first thing that comes to mind, you wouldn’t be very good. GPT-4 thinks for a few hundred tokens. If I think at 100 tokens a minute, that’s like what GPT-4 does. It’s equivalent to me thinking for three minutes. Suppose GPT-4 could think for millions of tokens. That’s +4 OOMs on test time compute on one problem. It can’t do it now. It gets stuck. It writes some code. It can do a little bit of iterative debugging, but eventually gets stuck and can’t correct its errors. There’s a big overhang. In other areas of ML, there’s a great paper on AlphaGo, where you can trade off train time and test time compute. If you can use 4 OOMs more test time compute, that’s almost like a 3.5x OOM bigger model. Again, if it’s 100 tokens a minute, a few million tokens is a few months of working time. There’s a lot more you can do in a few months of working time than just getting an answer right now. The question is how hard is it to unlock that? In the short timelines AI world, it’s not that hard. The reason it might not be that hard is that there are only a few extra tokens to learn. You need to learn things like error correction tokens where you’re like “ah, I made a mistake, let me think about that again.” You need to learn planning tokens where it’s like “I’m going to start by making a plan. Here’s my plan of attack. I’m going to write a draft and now I’m going to critique my draft and think about it.” These aren’t things that models can do now, but the question is how hard it is. There are two paths to agents. When Sholto was on your podcast, he talked about scaling leading to more nines of reliability. That’s one path. The other path is the unhobbling path. It needs to learn this System 2 process. If it can learn that, it can use millions of tokens and think coherently. Here’s an analogy. When you drive, you’re on autopilot most of the time. Sometimes you hit a weird construction zone or intersection. Sometimes my girlfriend is in the passenger seat and I’m like “ah, be quiet for a moment, I need to figure out what’s going on.” You go from autopilot to System 2 and you’re thinking about how to do it. Scaling improves that System 1 autopilot. The brute force way to get to agents is improving that system. If you can get System 2 working, you can quickly jump to something more agentified and test time compute overhang is unlocked. What’s the reason to think this is an easy win? Is there some loss function that easily enables System 2 thinking? There aren’t many animals with System 2 thinking. It took a long time for evolution to give us System 2 thinking. Pre-training has trillions of tokens of Internet text, I get that. You match that and get all of these free training capabilities. What’s the reason to think this is an easy unhobbling? First of all, pre-training is magical. It gave us a huge advantage for models of general intelligence because you can predict the next token. But there’s a common misconception. Predicting the next token lets the model learn incredibly rich representations. Representation learning properties are the magic of deep learning. Rather than just learning statistical artifacts, the models learn models of the world. That’s why they can generalize, because it learned the right representations. When you train a model, you have this raw bundle of capabilities that’s useful. The unhobbling from GPT-2 to GPT-4 took this raw mass and RLHF’d it into a good chatbot. That was a huge win. In the original InstructGPT paper, comparing RLHF vs. non-RLHF models it’s like a 100x model size win on human preference rating. It started to be able to do simple chain-of-thought and so on. But you still have this advantage of all these raw capabilities, and there’s still a huge amount you’re not doing with them. This pre-training advantage is also the difference to robotics. People used to say it was a hardware problem. The hardware is getting solved, but you don’t have this huge advantage of bootstrapping with pre-training. You don’t have all this unsupervised learning you can do. You have to start right away with RL self-play. The question is why RL and unhobbling might work. Bootstrapping is an advantage. Your Twitter bio is being pre-trained. You’re not being pre-trained anymore. You were pre-trained in grade school and high school. At some point, you transition to being able to learn by yourself. You weren’t able to do it in elementary school. High school is probably where it started and by college, if you’re smart, you can teach yourself. Models are just starting to enter that regime. It’s a little bit more scaling and then you figure out what goes on top. It won’t be trivial. A lot of deep learning seems obvious in retrospect. There’s some obvious cluster of ideas. There are some ideas that seem a little dumb but work. There are a lot of details you have to get right. We’re not going to get this next month. It’ll take a while to figure out. A while for you is like half a year. I don’t know, between six months and three years. But it’s possible. It’s also very related to the issue of the data wall. Here’s one intuition on learning by yourself. Pre-training is kind of like the teacher lecturing to you and the words are flying by. You’re just getting a little bit from it. That’s not what you do when you learn by yourself. When you learn by yourself, say you’re reading a dense math textbook, you’re not just skimming through it once. Some wordcels just skim through and reread and reread the math textbook and they memorize. What you do is you read a page, think about it, have some internal monologue going on, and have a conversation with a study buddy. You try a practice problem and fail a bunch of times. At some point it clicks, and you’re like, “this made sense.” Then you read a few more pages. We’ve kind of bootstrapped our way to just starting to be able to do that now with models. The question is, can you use all this sort of self-play, synthetic data, RL to make that thing work. Right now, there’s in-context learning, which is super sample efficient. In the Gemini paper, it just learns a language in-context. Pre-training, on the other hand, is not at all sample efficient. What humans do is a kind of in-context learning. You read a book, think about it, until eventually it clicks. Then you somehow distill that back into the weights. In some sense, that’s what RL is trying to do. RL is super finicky, but when it works it’s kind of magical. It’s the best possible data for the model. It’s when you try a practice problem, fail, and at some point figure it out in a way that makes sense to you. That’s the best possible data for you because it’s the way you would have solved the problem, rather than just reading how somebody else solved the problem, which doesn’t initially click. By the way, if that take sounds familiar it’s because it was part of the question I asked John Schulman. It goes to illustrate the thing I said in the intro. A bunch of the things I’ve learned about AI comes from these dinners we do before the interviews with me, you, Sholto, and a couple of others. We’re like, “what should I ask John Schulman, what I should ask Dario.” Suppose this is the way things go and we get these unhobblings— And the scaling. You have this baseline of this enormous force of scaling. GPT-2 was amazing. It could string together plausible sentences, but it could barely do anything. It was kind of like a preschooler. GPT-4, on the other hand, could write code and do hard math, like a smart high schooler. This big jump in capability is explored in the essay series. I count the orders of magnitude of compute and scale-up of algorithmic progress. Scaling alone by 2027-2028 is going to do another preschool to high school jump on top of GPT-4. At a per token level, the models will be incredibly smart. They’ll gain more reliability, and with the addition of unhobblings, they’ll look less like chatbots and more like agents or drop-in remote workers. That’s when things really get going. I want to ask more questions about this but let’s zoom out. Suppose you’re right about this. This is because of the 2027 cluster which is at 10 GW? 2028 is 10 GW. Maybe it’ll be pulled forward. Something like a 5.5 level by 2027, whatever that’s called. What does the world look like at that point? You have these remote workers who can replace people. What is the reaction to that in terms of the economy, politics, and geopolitics? 2023 was a really interesting year to experience as somebody who was really following the AI stuff. What were you doing in 2023? OpenAI. When you were at OpenAI in 2023, it was a weird thing. You almost didn’t want to talk about AI or AGI. It was kind of a dirty word. Then in 2023, people saw ChatGPT for the first time, they saw GPT-4, and it just exploded. It triggered huge capital expenditures from all these firms and an explosion in revenue from Nvidia and so on. Things have been quiet since then, but the next thing has been in the oven. I expect every generation these g-forces to intensify. People will see the models. They won’t have counted the OOMs so they’re going to be surprised. It’ll be kind of crazy. Revenue is going to accelerate. Suppose you do hit $10 billion by the end of this year. Suppose it just continues on the trajectory of revenue doubling every six months. It’s not actually that far from $100 billion, maybe by 2026. At some point, what happened to Nvidia is going to happen to Big Tech. It’s going to explode. A lot more people are going to feel it. 2023 was the moment for me where AGI went from being this theoretical, abstract thing. I see it, I feel it, and I see the path. I see where it’s going. I can see the cluster it’s trained on, the rough combination of algorithms, the people, how it’s happening. Most of the world is not there yet. Most of the people who feel it are right here. A lot more of the world is going to start feeling it. That’s going to start being intense. Right now, who feels it? You can go on Twitter and there are these GPT wrapper companies, like, “whoa, GPT-4 is going to change our business.” I’m so bearish on the wrapper companies because they’re betting on stagnation. They’re betting that you have these intermediate models and it takes so much schlep to integrate them. I’m really bearish because we’re just going to sonic boom you. We’re going to get the unhobblings. We’re going to get the drop-in remote worker. Your stuff is not going to matter. So that’s done. SF, this crowd, is paying attention now. Who is going to be paying attention in 2026 and 2027? Presumably, these are years in which hundreds of billions of capex is being spent on AI. The national security state is going to start paying a lot of attention. I hope we get to talk about that. Let’s talk about it now. What happens? What is the immediate political reaction? Looking internationally, I don’t know if Xi Jinping sees the GPT-4 news and goes, “oh, my God, look at the MMLU score on that. What are we doing about this, comrade?” So what happens when he sees a remote worker replacement and it has $100 billion in revenue? There’s a lot of businesses that have $100 billion in revenue, and people aren’t staying up all night talking about it. The question is, when does the CCP and when does the American national security establishment realize that superintelligence is going to be absolutely decisive for national power? This is where the intelligence explosion stuff comes in, which we should talk about later. You have AGI. You have this drop-in remote worker that can replace you or me, at least for remote jobs. Fairly quickly, you turn the crank one or two more times and you get a thing that’s smarter than humans. Even more than just turning the crank a few more times, one of the first jobs to be automated is going to be that of an AI researcher or engineer. If you can automate AI research, things can start going very fast. Right now, there’s already at this trend of 0.5 OOMs a year of algorithmic progress. At some point, you’re going to have GPU fleets in the tens of millions for inference or more. You’re going to be able to run 100 million human equivalents of these automated AI researchers. If you can do that, you can maybe do a decade’s worth of ML research progress in a year. You get some sort of 10x speed up. You can make the jump to AI that is vastly smarter than humans within a year, a couple of years. That broadens from there. You have this initial acceleration of AI research. You apply R&D to a bunch of other fields of technology. At this point, you have a billion super intelligent researchers, engineers, technicians, everything. They’re superbly competent at all things. They’re going to figure out robotics. We talked about that being a software problem. Well, you have a billion super smart — smarter than the smartest human researchers — AI researchers in your cluster. At some point during the intelligence explosion, they’re going to be able to figure out robotics. Again, that’ll expand. If you play this picture forward, it is fairly unlike any other technology. A couple years of lead could be utterly decisive in say, military competition. If you look at the first Gulf War, Western coalition forces had a 100:1 kill ratio. They had better sensors on their tanks. They had better precision missiles, GPS, and stealth. They had maybe 20-30 years of technological lead. They just completely crushed them. Superintelligence applied to broad fields of R&D — and the industrial explosion that comes from it, robots making a lot of material — could compress a century’s worth of technological progress into less than a decade. That means that a couple years could mean a Gulf War 1-style advantage in military affairs. That’s including a decisive advantage that even preempts nukes. How do you find nuclear stealth submarines? Right now, you have sensors and software to detect where they are. You can do that. You can find them. You have millions or billions of mosquito-sized drones, and they take out the nuclear submarines. They take out the mobile launchers. They take out the other nukes. It’s potentially enormously destabilizing and enormously important for national power. At some point people are going to realize that. Not yet, but they will. When they do, it won’t just be the AI researchers in charge. The CCP is going to have an all-out effort to infiltrate American AI labs. It’ll involve billions of dollars, thousands of people, and the full force of the Ministry of State Security. The CCP is going to try to outbuild us. They added as much power in the last decade as an entire US electric grid. So the 100 GW cluster, at least the 100 GW part of it, is going to be a lot easier for them to get. By this point, it’s going to be an extremely intense international competition. One thing I’m uncertain about in this picture is if it’s like what you say, where it’s more of an explosion. You’ve developed an AGI. You make it into an AI researcher. For a while, you’re only using this ability to make hundreds of millions of other AI researchers. The thing that comes out of this really frenetic process is a superintelligence. Then that goes out in the world and is developing robotics and helping you take over other countries and whatever. It’s a little bit more gradual. It’s an explosion that starts narrowly. It can do cognitive jobs. The highest ROI use for cognitive jobs is to make the AI better and solve robotics. As you solve robotics, now you can do R&D in biology and other technology. Initially, you start with the factory workers. They’re wearing the glasses and AirPods, and the AI is instructing them because you can make any worker into a skilled technician. Then you have the robots come in. So this process expands. Meta’s Ray-Bans are a complement to Llama. With the fabs in the US, their constraint is skilled workers. Even if you don’t have robots, you have the cognitive superintelligence and can kind of make them all into skilled workers immediately. That’s a very brief period. Robots will come soon. Suppose this is actually how the tech progresses in the United States, maybe because these companies are already generating hundreds of billions of dollars of AI revenue At this point, companies are borrowing hundreds of billions or more in the corporate debt markets. Why is a CCP bureaucrat, some 60-year-old guy, looking at this and going, “oh, Copilot has gotten better now” and now— This is much more than Copilot has gotten better now. It’d require shifting the production of an entire country, dislocating energy that is otherwise being used for consumer goods or something, and feeding all that into the data centers. Part of this whole story is that you realize superintelligence is coming soon. You realize it and maybe I realize it. I’m not sure how much I realize it. Will the national security apparatus in the United States and the CCP realize it? This is a really key question. We have a few more years of mid-game. We have a few more 2023s. That just starts updating more and more people. The trend lines will become clear. You will see some amount of the COVID dynamic. COVID in February of 2020 honestly feels a lot like today. It feels like this utterly crazy thing is coming. You see the exponential and yet most of the world just doesn’t realize it. The mayor of New York is like, “go out to the shows,” and “this is just Asian racism.” At some point, people saw it and then crazy, radical reactions came. By the way, what were you doing during COVID? Was it your freshman or sophomore year? Junior. Still, you were like a 17-year-old junior or something right? Did you short the market or something? Did you sell at the right time? Yeah. So there will be a March 2020 moment. You can make the analogy you make in the series that this will cause a reaction like, “we have to do the Manhattan Project again for America here.” I wonder what the politics of this will be like. The difference here is that it’s not just like, “we need the bomb to beat the Nazis.” We’ll be building this thing that makes all our energy prices go up a bunch and it’s automating a lot of our jobs. The climate change stuff people are going to be like, “oh, my God, it’s making climate change worse and it’s helping Big Tech.” Politically, this doesn’t seem like a dynamic where the national security apparatus or the president is like, “we have to step on the gas here and make sure America wins.” Again, a lot of this really depends on how much people are feeling it and how much people are seeing it. Our generation is so used to peace, American hegemony and nothing matters. The historical norm is very much one of extremely intense and extraordinary things happening in the world with intense international competition. There’s a 20-year very unique period. In World War II, something like 50% of GDP went to war production. The US borrowed over 60% of GDP. With Germany and Japan I think it was over 100%. In World War I, the UK, France, and Germany all borrowed over 100% of GDP. Much more was on the line. People talk about World War I being so destructive with 20 million Soviet soldiers dying and 20% of Poland. That happened all the time. During the Seven Years’ War something like 20-30% of Prussia died. In the Thirty Years’ War, up to 50% of a large swath of Germany died. Will people see that the stakes here are really high and that history is actually back? The American national security state thinks very seriously about stuff like this. They think very seriously about competition with China. China very much thinks of itself on this historical mission of the rejuvenation of the Chinese nation. They think a lot about national power. They think a lot about the world order. There’s a real question on timing. Do they start taking this seriously when the intelligence explosion is already happening quite late. Do they start taking this seriously two years earlier? That matters a lot for how things play out. At some point they will and they will realize that this will be utterly decisive for not just some proxy war but for major questions. Can liberal democracy continue to thrive? Can the CCP continue existing? That will activate forces that we haven’t seen in a long time. The great power conflict definitely seems compelling. All kinds of different things seem much more likely when you think from a historical perspective. You zoom out beyond the liberal democracy that we’ve had the pleasure to live in America for say the last 80 years. That includes things like dictatorships, war, famine, etc. I was reading The Gulag Archipelago and one of the chapters begins with Solzhenitsyn saying how if you had told a Russian citizen under the tsars that because of all these new technologies — we wouldn’t see some Great Russian revival with Russia becoming a great power and the citizens made wealthy — you would see tens of millions of Soviet citizens tortured by millions of beasts in the worst possible ways. If you’d told them that that would be the result of the 20th century, they wouldn’t have believed you. They’d have called you a slanderer. The possibilities for dictatorship with superintelligence are even crazier as well. Imagine you have a perfectly loyal military and security force. No more rebellions. No more popular uprisings. You have perfect lie detection. You have surveillance of everybody. You can perfectly figure out who’s the dissenter and weed them out. No Gorbachev who had some doubts about the system would have ever risen to power. No military coup would have ever happened. There’s a real way in which part of why things have worked out is that ideas can evolve. There’s some sense in which time heals a lot of wounds and solves a lot of debates. Throughout time, a lot of people had really strong convictions, but a lot of those have been overturned over time because there’s been continued pluralism and evolution. Imagine applying a CCP-like approach to truth where truth is what the party says. When you supercharge that with superintelligence, that could just be locked in and enshrined for a long time. The possibilities are pretty terrifying. To your point about history and living in America for the past eight years, this is one of the things I took away from growing up in Germany. A lot of this stuff feels more visceral. My mother grew up in the former East, my father in the former West. They met shortly after the Wall fell. The end of the Cold War was this extremely pivotal moment for me because it’s the reason I exist. I grew up in Berlin with the former Wall. My great-grandmother, who is still alive, is very important in my life. She was born in 1934 and grew up during the Nazi era. In World War II, she saw the firebombing of Dresden from this country cottage where they were as kids. Then she spent most of her life in the East German communist dictatorship. She’d tell me about how Soviet tanks came when there was the popular uprising in 1954. Her husband was telling her to get home really quickly and get off the streets. She had a son who tried to ride a motorcycle across the Iron Curtain and then was put in a Stasi prison for a while. Finally, when she’s almost 60, it was the first time she lived in a free country, and a wealthy country. When I was a kid, the thing she always really didn’t want me to do was get involved in politics. Joining a political party had very bad connotations for her. She raised me when I was young. So it doesn’t feel that long ago. It feels very close. There’s one thing I wonder about when we’re talking today about the CCP. The people in China who will be doing their version of this project will be AI researchers who are somewhat Westernized. They’ll either have gotten educated in the West or have colleagues in the West. Are they going to sign up for the CCP project that’s going to hand over control to Xi Jinping? What’s your sense of that? Fundamentally, they’re just people, right? Can’t you convince them about the dangers of superintelligence? Will they be in charge though? In some sense, this is also the case in the US. This is like the rapidly depreciating influence of the lab employees. Right now, the AI lab employees have so much power. You saw this November event. It’s so much power. Both are going to get automated and they’re going to lose all their power. It’ll just be a few people in charge with their armies of automated AIs. It’s also the politicians and the generals and the national security state. There are some of these classic scenes from the Oppenheimer movie. The scientists built it and then the bomb was shipped away and it was out of their hands. It’s good for lab employees to be aware of this. You have a lot of power now, but maybe not for that long. Use it wisely. I do think they would benefit from some more organs of representative democracy. What do you mean by that? In the OpenAI board events, employee power is exercised in a very direct democracy way. How some of that went about really highlighted the benefits of representative democracy and having some deliberative organs. Interesting. Let’s go back to the $100 billion revenue question. The companies are trying to build clusters that are this big. Where are they building it? Say it’s the amount of energy that would be required for a small or medium-sized US state. Does Colorado then get no power because it’s happening in the United States? Is it happening somewhere else? This is the thing that I always find funny, when you talk about Colorado getting no power. The easy way to get the power would be to displace less economically useful stuff. Buy up the aluminum smelting plant that has a gigawatt. We’re going to replace it with the data center because that’s important. That’s not actually happening because a lot of these power contracts are really locked in long-term. Also, people don’t like things like this. In practice what it requires, at least right now, is building new power. That might change. That’s when things get really interesting, when it’s like, “no, we’re just dedicating all of the power to the AGI.” So right now it’s building new power. 10 GW is quite doable. It’s like a few percent of US natural gas production. When you have the 10 GW training cluster, you have a lot more inference. 100 gigawatts is where it starts getting pretty wild. That’s over 20% of US electricity production. It’s pretty doable, especially if you’re willing to go for natural gas. It is incredibly important that these clusters are in the United States. Why does it matter that it’s in the US? There are some people who are trying to build clusters elsewhere. There’s a lot of free-flowing Middle Eastern money that’s trying to build clusters elsewhere. This comes back to the national security question we talked about. Would you do the Manhattan Project in the UAE? You can put the clusters in the US and you can put them in allied democracies. Once you put them in authoritarian dictatorships, you create this irreversible security risk. Once the cluster is there, it’s much easier for them to exfiltrate the weights. They can literally steal the AGI, the superintelligence. It’s like they got a direct copy of the atomic bomb. It makes it much easier for them. They have weird ties to China. They can ship that to China. That’s a huge risk. Another thing is they can just seize the compute. The issue here is people right now are thinking of this as ChatGPT, Big Tech product clusters. The clusters being planned now, three to five years out, may well be the AGI, superintelligence clusters. When things get hot, they might just seize the compute. Suppose we put 25% of the compute capacity in these Middle Eastern dictatorships. Say they seize that. Now it’s a ratio of compute of 3:1. We still have more, but even with only 25% of compute there it starts getting pretty hairy. 3:1 is not that great of a ratio. You can do a lot with that amount of compute. Say they don’t actually do this. Even if they don’t actually seize the compute, even if they actually don’t steal the weights, there’s just a lot of implicit leverage you get. They get seats at the AGI table. I don’t know why we’re giving authoritarian dictatorships the seat at the AGI table. There’s going to be a lot of compute in the Middle East if these deals go through. First of all, who is it? Is it just every single Big Tech company trying to figure it out over there? It’s not everybody, some. There are reports, I think Microsoft. We’ll get into it. So say the UAE gets a bunch of compute because we’re building the clusters there. Let’s say they have 25% of the compute. Why does a compute ratio matter? If it’s about them being able to kick off the intelligence explosion, isn’t it just some threshold where you have 100 million AI researchers or you don’t? You can do a lot with 33 million extremely smart scientists. That might be enough to build the crazy bio weapons. Then you’re in a situation where they stole the weights and they seized the compute. Now they can make these crazy new WMDs that will be possible with superintelligence. Now you’ve just proliferated the stuff that’ll be really powerful. Also, 3x on compute isn’t actually that much. The riskiest situation is if we’re in some sort of really neck and neck, feverish international struggle. Say we’re really close with the CCP and we’re months apart. The situation we want to be in — and could be in if we play our cards right — is a little bit more like the US building the atomic bomb versus the German project years behind. If we have that, we just have so much more wiggle room to get safety right. We’re going to be building these crazy new WMDs that completely undermine nuclear deterrence. That’s so much easier to deal with if you don’t have somebody right on your tails and you have to go at maximum speed. You have no wiggle room. You’re worried that at any time they can overtake you. They can also just try to outbuild you. They might literally win. China might literally win if they can steal the weights, because they can outbuild you. They may have less caution, both good and bad caution in terms of whatever unreasonable regulations we have. If you’re in this really tight race, this sort of feverish struggle, that’s when there’s the greatest peril of self-destruction. Presumably the companies that are trying to build clusters in the Middle East realize this. Is it just that it’s impossible to do this in America? If you want American companies to do this at all, do you have to do it in the Middle East or not at all? Then you just have China build a Three Gorges Dam cluster. There’s a few reasons. People aren’t thinking about this as the AGI superintelligence cluster. They’re just like, “ah, cool clusters for my ChatGPT.” If you’re doing ones for inference, presumably you could spread them out across the country or something. The ones they’re building, they’re going to do one training run in a single thing they’re building. It’s just hard to distinguish between inference and training compute. People can claim it’s inference compute, but they might realize that actually this is going to be useful for training compute too. Because of synthetic data and things like that? RL looks a lot like inference, for example. Or you just end up connecting them in time. It’s a lot like raw materials. It’s like placing your uranium refinement facilities there. So there are a few reasons. One, they don’t think about this as the AGI cluster. Another is just that there’s easy money coming from the Middle East. Another one is that some people think that you can’t do it in the US. We actually face a real system competition here. Some people think that only autocracies that can do this with top-down mobilization of industrial capacity and the power to get stuff done fast. Again, this is the sort of thing we haven’t faced in a while. But during the Cold War, there was this intense system competition. East vs. West Germany was this. It was West Germany as liberal democratic capitalism vs. state-planned communism. Now it’s obvious that the free world would win. But even as late as 1961, Paul Samuelson was predicting that the Soviet Union would outgrow the United States because they were able to mobilize industry better. So there are some people who shitpost about loving America, but then in private they’re betting against America. They’re betting against the liberal order. Basically, it’s just a bad bet. This stuff is really possible in the US. To make it possible in the US, to some degree we have to get our act together. There are basically two paths to doing it in the US. One is you just have to be willing to do natural gas. There’s ample natural gas. You put your cluster in West Texas. You put it in southwest Pennsylvania by the Marcellus Shale. The 10 GW cluster is super easy. The 100 GW cluster is also pretty doable. I think natural gas production in the United States has almost doubled in a decade. You do that one more time over the next seven years, you could power multiple trillion-dollar data centers. The issue there is that a lot of people made these climate commitments, not just the government. It’s actually the private companies themselves, Microsoft, Amazon, etc., that have made these climate commitments. So they won’t do natural gas. I admire the climate commitments, but at some point the national interest and national security is more important. The other path is doing green energy megaprojects. You do solar and batteries and SMRs and geothermal. If we want to do that, there needs to be a broad deregulatory push. You can’t have permitting take a decade. You have to reform FERC. You have to have blanket NEPA exemptions for this stuff. There are inane state-level regulations. You can build the solar panels and batteries next to your data center, but it’ll still take years because you actually have to hook it up to the state electrical grid. You have to use governmental powers to create rights of way to have multiple clusters and connect them and have the cables. Ideally we do both. Ideally we do natural gas and the broader deregulatory green agenda. We have to do at least one. Then this stuff is possible in the United States. Before the conversation I was reading a good book about World War II industrial mobilization in the United States called Freedom’s Forge. I’m thinking back on that period, especially in the context of reading Patrick Collison’s Fast and the progress study stuff. There’s this narrative out there that we had state capacity back then and people just got shit done but that now it’s a clusterfuck. It wasn’t at all the case! It was really interesting. You had people from the Detroit auto industry side, like William Knudsen, who were running mobilization for the United States. They were extremely competent. At the same time you had labor organization and agitation, which is very analogous to the climate change pledges and concerns we have today. They would literally have these strikes, into 1941, costing millions of man-hours worth of time when we’re trying to make tens of thousands of planes a month. They would just debilitate factories for trivial concessions from capital that were pennies on the dollar. There were concerns that the auto companies were trying to use the pretext of a potential war to prevent paying labor the money it deserves. So with what climate change is today, you might think, “ah, America’s fucked. We’re not going to be able to build this shit if you look at NEPA or something,” I didn’t realize how debilitating labor was in World War II. It wasn’ just that. Before 1939, the American military was in total shambles. You read about it and it reads a little bit like the German military today. Military expenditures were I think less than 2% of GDP. All the European countries had gone, even in peacetime, above 10% of GDP. It was rapid mobilization starting from nothing. We were making no planes. There were no military contracts. Everything had been starved during the Great Depression. But there was this latent capacity. At some point the United States got its act together. This applies the other way around too with China. Sometimes people count them out a little bit with the export controls and so on. They’re able to make 7-nanometer chips now. There’s a question of how many they could make. There’s at least a possibility that they’re going to mature that ability and make a lot of 7-nanometer chips. There’s a lot of latent industrial capacity in China. They are able to build a lot of power fast. Maybe that isn’t activated for AI yet. At some point, the same way the United States and a lot of people in the US government are going to wake up, the CCP is going to wake up. Companies realize that scaling is a thing. Obviously their whole plans are contingent on scaling. So they understand that in 2028 we’re going to be building 10 GW data centers. At that point, the people who can keep up are Big Tech, potentially at the edge of their capabilities, sovereign wealth fund-funded things, and also major countries like America and China. What’s their plan? With the AI labs, what’s their plan given this landscape? Do they not want the leverage of being in the United States? The Middle East does offer capital, but America has plenty of capital. We have trillion-dollar companies. What are these Middle Eastern states? They’re kind of like trillion-dollar oil companies. We have trillion-dollar companies and very deep financial markets. Microsoft could issue hundreds of billions of dollars of bonds and they can pay for these clusters. Another argument being made, which is worth taking seriously, is that if we don’t work with the UAE or with these Middle Eastern countries, they’re just going to go to China. They’re going to build data centers and pour money into AI regardless. If we don’t work with them, they’ll just support China. There’s some merit to the argument in the sense that we should be doing benefit-sharing with them. On the road to AGI, there should be two tiers of coalitions. There should be a narrow coalition of democracies that’s developing AGI. Then there should be a broader coalition of other countries, including dictatorships, and we should offer them some of the benefits of AI. If the UAE wants to use AI products, run Meta recommendation engines, or run the last-generation models, that’s fine. By default, they just wouldn’t have had this seat at the AGI table. So they have some money, but a lot of people have money. The only reason they’re getting this seat at the AGI table and giving these dictators this leverage over this extremely important national security technology, is because we’re getting them excited and offering it to them. Who specifically is doing this? Who are the companies who are going there to fundraise? It’s been reported that Sam Altman is trying to raise $7 trillion or whatever for a chip project. It’s unclear how many of the clusters will be there, but definitely stuff is happening. There’s another reason I’m a little suspicious of this argument that if the US doesn’t work with them, they’ll go to China. I’ve heard from multiple people — not from my time at OpenAI, and I haven’t seen the memo — that at some point several years ago, OpenAI leadership had laid out a plan to fund and sell AGI by starting a bidding war between the governments of the United States, China, and Russia. It’s surprising to me that they’re willing to sell AGI to the Chinese and Russian governments. There’s also something that feels eerily familiar about starting this bidding war and then playing them off each other, saying, “well, if you don’t do this, China will do it.” Interesting. That’s pretty fucked up. Suppose you’re right. We ended up in this place because, as one of our friends put it, the Middle East has billions or trillions of dollars up for persuasion like no other place in the world. With little accountability. There’s no Microsoft board. It’s only the dictator. Let’s say you’re right, that you shouldn’t have gotten them excited about AGI in the first place. Now we’re in a place where they are excited about AGI and they’re like, “fuck, we want to have GPT-5 while you’re going to be off building superintelligence. This Atoms for Peace thing doesn’t work for us.” If you’re in this place, don’t they already have the leverage? The UAE on its own is not competitive. They’re already export-controlled. You’re not supposed to ship Nvidia chips over there. It’s not like they have any of the leading AI labs. They have money, but it’s hard to just translate money into progress. But I want to go back to other things you’ve been saying in laying out your vision. There’s this almost industrial process of putting in the compute and algorithms, adding that up, and getting AGI on the other end. If it’s something more like that, then the case for somebody being able to catch up rapidly seems more compelling than if it’s some bespoke… Well, if they can steal the algorithms and if they can steal the weights, that’s really important. How easy would it be for an actor to steal the things that are not the trivial released things, like Scarlett Johansson’s voice, but the RL things we’re talking about, the unhobblings? It’s all extremely easy. They don’t make the claim that it’s hard. DeepMind put out their Frontier Safety Framework and they lay out security levels, zero to four. Four is resistant to state activity. They say, we’re at level zero. Just recently, there was an indictment of a guy who stole a bunch of really important AI code and went to China with it. All he had to do to steal the code was copy it, put it into Apple Notes, and export it as a PDF. That got past their monitoring. Google has the best security of any of the AI labs probably, because they have the Google infrastructure. I would think of the security of a startup. What does security of a startup look like? It’s not that good. It’s easy to steal. Even if that’s the case, a lot of your post is making the argument for why we are going to get the intelligence explosion. If we have somebody with the intuition of an Alec Radford to come up with all these ideas, that intuition is extremely valuable and you can scale that up. If it’s just intuition, then that’s not going to be just in the code, right? Also because of export controls, these countries are going to have slightly different hardware. You’re going to have to make different trade-offs and probably rewrite things to be compatible with that. Is it just a matter of getting the right pen drive and plugging it into the gigawatt data center next to the Three Gorges Dam and then you’re off to the races? There are a few different things, right? One threat model is just them stealing the weights themselves. The weights one is particularly insane because they can just steal the literal end product — just make a replica of the atomic bomb — and then they’re ready to go. That one is extremely important around the time we have AGI and superintelligence because China can build a big cluster by default. We’d have a big lead because we have the better scientists, but if we make the superintelligence and they just steal it, they’re off to the races. Weights are a little bit less important right now because who cares if they steal the GPT-4 weights. We still have to get started on weight security now because if we think there’s AGI by 2027, this stuff is going to take a while. It’s not just going to be like, “oh, we do some access control.” If you actually want to be resistant to Chinese espionage, it needs to be much more intense. The thing that people aren’t paying enough attention to is the secrets. The compute stuff is sexy, but people underrate the secrets. The half an order of magnitude a year is just by default, sort of algorithmic progress. That’s huge. If we have a few years of lead, by default, that’s a 10-30x, 100x bigger cluster, if we protect them. There’s this additional layer of the data wall. We have to get through the data wall. That means we actually have to figure out some sort of basic new paradigm. So it’s the “AlphaGo step two.” “AlphaGo step one” learns from human imitation. “AlphaGo step two” is the kind of self-play RL thing that everyone’s working on right now. Maybe we’re going to crack it. If China can’t steal that, then they’re stuck. If they can steal it, they’re off to the races. Whatever that thing is, can I literally write it down on the back of a napkin? If it’s that easy, then why is it so hard for them to figure it out? If it’s more about the intuitions, then don’t you just have to hire Alec Radford? What are you copying down? There are a few layers to this. At the top is the fundamental approach. On pre-training it might be unsupervised learning, next token prediction, training on the entire Internet. You actually get a lot of juice out of that already. That one’s very quick to communicate. Then there’s a lot of details that matter, and you were talking about this earlier. It’s probably going to be somewhat obvious in retrospect, or there’s going to be some not too complicated thing that’ll work, but there’s going to be a lot of details to get that. If that’s true, then again, why do we think that getting state-level security in these startups will prevent China from catching up? It’s just like, “oh, we know some sort of self-play RL will be required to get past the data wall.” It’s going to be solved by 2027, right? It’s not that hard. The US, and the leading labs in the United States, have this huge lead. By default, China actually has some good LLMs because they’re just using open source code, like Llama. People really underrate both the divergence on algorithmic progress and the lead the US would have by default because all this stuff was published until recently. Look at Chinchilla Scaling laws, MoE papers, transformers. All that stuff was published. That’s why open source is good and why China can make some good models. Now, they’re not publishing it anymore. If we actually kept it secret, it would be a huge edge. To your point about tacit knowledge and Alec Radford, there’s another layer at the bottom that is something about large-scale engineering work to make these big training runs work. That is a little bit more like tacit knowledge, but China will be able to figure that out. It’s engineering schlep, and they’re going to figure out how to do it. Why can’t they figure that out, but not how to get the RL thing working? I don’t know. Germany during World War II went down the wrong path with heavy water. There’s an amazing anecdote in The Making of the Atomic Bomb about this. Secrecy was one of the most contentious issues early on. Leo Szilard really thought a nuclear chain reaction and an atomic bomb were possible. He went around saying, “this is going to be of enormous strategic and military importance.” A lot of people didn’t believe it or thought, “maybe this is possible, but I’m going to act as though it’s not, and science should be open.” In the early days, there had been some incorrect measurements made on graphite as a moderator. Germany thought graphite wasn’t going to work, so they had to do heavy water. But then Enrico Fermi made new measurements indicating that graphite would work. This was really important. Szilard assaulted Fermi with another secrecy appeal and Fermi was pissed off, throwing a temper tantrum. He thought it was absurd, saying, “come on, this is crazy.” But Szilard persisted, and they roped in another guy, George Pegram. In the end, Fermi didn’t publish it. That was just in time. Fermi not publishing meant that the Nazis didn’t figure out graphite would work. They went down the path of heavy water, which was the wrong path. This is a key reason why the German project didn’t work out. They were way behind. We face a similar situation now. Are we just going to instantly leak how to get past the data wall and what the next paradigm is? Or are we not? The reason this would matter is if being one year ahead would be a huge advantage. In the world where you deploy AI over time they’re just going to catch up anyway. I interviewed Richard Rhodes, the guy who wrote The Making of the Atomic Bomb. One of the anecdotes he had was when the Soviets realized America had the bomb. Obviously, we dropped it in Japan. Lavrentiy Beria — the guy who ran the NKVD, a famously ruthless and evil guy — goes to the Soviet scientist who was running their version of the Manhattan Project. He says, “comrade, you will get us the American bomb.” The guy says, “well, listen, their implosion device actually is not optimal. We should make it a different way.” Beria says, “no, you will get us the American bomb, or your family will be camp dust.” The thing that’s relevant about that anecdote is that the Soviets would have had a better bomb if they hadn’t copied the American design, at least initially. That suggests something about history, not just for the Manhattan Project. There’s often this pattern of parallel invention because the tech tree implies that a certain thing is next — in this case, a self-play RL — and people work on that and are going to figure it out around the same time. There’s not going to be that much gap in who gets it first. Famously, a bunch of people invented the light bulb around the same time. Is it the case that it might be true but the one year or six months makes the difference? Two years makes all the difference. I don’t know if it’ll be two years though. If we lock down the labs, we have much better scientists. We’re way ahead. It would be two years. Even six months, a year, would make a huge difference. This gets back to the intelligence explosion dynamics. A year might be the difference between a system that’s sort of human-level and a system that is vastly superhuman. It might be like five OOMs. Look at the current pace. Three years ago, on the math benchmark — these are really difficult high school competition math problems — we were at a few percent, we couldn’t solve anything. Now it’s solved. That was at the normal pace of AI progress. You didn’t have a billion superintelligent researchers. A year is a huge difference, particularly after superintelligence. Once this is applied to many elements of R&D, you get an industrial explosion with robots and other advanced technologies. A couple of years might yield decades worth of progress. Again, it’s like the technological lead the U.S. had in the first Gulf War, when the 20-30 years of technological lead proved totally decisive. It really matters. Here’s another reason it really matters. Suppose they steal the weights, suppose they steal the algorithms, and they’re close on our tails. Suppose we still pull out ahead. We’re a little bit faster and we’re three months ahead. The world in which we’re really neck and neck, we only have a three-month lead, is incredibly dangerous. We’re in this feverish struggle where if they get ahead, they get to dominate, maybe they get a decisive advantage. They’re building clusters like crazy. They’re willing to throw all caution to the wind. We have to keep up. There are crazy new WMDs popping up. Then we’re going to be in the situation where it’s crazy new military technology, crazy new WMDs, deterrence, mutually assured destruction keeps changing every few weeks. It’s a completely unstable, volatile situation that is incredibly dangerous. So you have to look at it from the point of view that these technologies are dangerous, from the alignment point of view. It might be really important during the intelligence explosion to have a six-month wiggle room to be like, “look, we’re going to dedicate more compute to alignment during this period because we have to get it right. We’re feeling uneasy about how it’s going.” One of the most important inputs to whether we will destroy ourselves or whether we will get through this incredibly crazy period is whether we have that buffer. Before we go further, it’s very much worth noting that almost nobody I talk to thinks about the geopolitical implications of AI. I have some object-level disagreements that we’ll get into, things I want to iron out. I may not disagree in the end. The basic premise is that if you keep scaling, if people realize that this is where intelligence is headed, it’s not just going to be the same old world. It won’t just be about what model we’re deploying tomorrow or what the latest thing is. People on Twitter are like, “oh, GPT-4 is going to shake your expectations” or whatever. COVID is really interesting because when March 2020 hit, it became clear to the world — presidents, CEOs, media, the average person — that there are other things happening in the world right now but the main thing we as a world are dealing with right now is COVID. Soon it will be AGI. This is the quiet period. Maybe you want to go on vacation. Maybe now is the last time you can have some kids. My girlfriend sometimes complains when I’m off doing work that I don’t spend enough time with her. She threatens to replace me with GPT-6 or whatever. I’m like, “GPT-6 will also be too busy doing AI research.” Why aren’t other people talking about national security? I made this mistake with COVID. In February of 2020, I thought it was going to sweep the world and all the hospitals would collapse. It would be crazy, and then it’d be over. A lot of people thought this kind of thing at the beginning of COVID. They shut down their office for a month or whatever. The thing I just really didn’t price in was societal reaction. Within weeks, Congress spent over 10% of GDP on COVID measures. The entire country was shut down. It was crazy. I didn’t sufficiently price it in with COVID. Why do people underrate it? Being in the trenches actually gives you a less clear picture of the trend lines. You don’t have to zoom out that much, only a few years. When you’re in the trenches, you’re trying to get the next model to work. There’s always something that’s hard. You might underrate algorithmic progress because you’re like, “ah, things are hard right now,” or “data wall”