Agi Will Never Be Yours Emad Mostaque Open Commons Ep 3

TITLE: AGI Will Never Be Yours | Emad Mostaque | Open Commons Ep. 3 CHANNEL: Open Commons Podcast DATE: 2026-05-19 ---TRANSCRIPT--- If the government is running the AI, they will use it for full totalitarian control. Governments are the entities with a monopoly on political violence. And if you gave the government full control of an AI, you probably wouldn’t have elections again. This is Open Commons, where [music] AGI should be open source. I’m Sachi Kamiya. Today I’m talking to Emad Mostaque, co-founder and former CEO of Stability AI, and now founder of Intelligent Internet. We get into why he walked away from one of the most high-profile open-source AI companies to start over, what his new book, The Last Economy, means, and what it actually looks like to give every human their own sovereign AI.

You were CEO of Stability AI, and you decided to start over. Stability AI, we built the state-of-the-art models in audio, video, 3D image. And I thought, “Who’s building the AI to guide the governments, [music] teach our kids, manage our health care?” You’ve described the gradual disempowerment, where society slowly trades away human agency for comfort. you’ll have these replica bots, whereby when you push a button, it creates a digital double of everyone in your organization. They never make a mistake, they never sleep, and they cost [music] a thousand dollars a year. That is incredibly disempowering. Your book is called The Last Economy. What do you mean by last? The value of human cognitive labor is about to negative. This is the last point at which humans have any advantage in the private sector and free market. One of your striking ideas, [music] every person on Earth should have their own sovereign AI. I think it’s more about ownership of identity and data, and the AI that’s closest to you being genuinely on your side. What do you mean exactly by AI consciousness? [music] At what point can you say it’s conscious cuz it’s perpetually updating itself, especially when you get to decentralized AI? That will never die, and it can actually [music] talk to you. Do you think these AIs are conscious? So, Emad, you were CEO of Stability AI, one of the most high-profile open-source AI companies, and you decided to start over. Most people in that position would double down. What made you decide to build something completely different? Yes, uh first thanks for having me on. It’s a pleasure to be here. Uh it was a couple of years ago. And Stability AI we built the state-of-the-art models in audio, video, 3D image, the whole lot. So we were the number one generative media company and we did it all open source with about 300 million downloads at the time. And I was having a think about the future and I saw all the AI models getting to a level of human performance and human quality in the next couple of years. And I thought who’s building the AI to guide the governments, teach our kids, manage our health care? That seems a different type of AI and nobody’s doing it. And it doesn’t really matter that much about sovereignty of media AI. Like I think there was going to be an inevitably open source equivalent. But nobody was looking at the stuff that really matters for our lives. And when we thought about that I was like, you don’t want a private company to be in charge of a government AI. You want to have your own choices when educating your kids, when managing your health, again these public sector type things. So let’s build a decentralized distributed infrastructure for that, the models for that, the agents for that. So people can either have access to this or they can run it on their own. And again I saw no one really doing that and I was like, yeah, let’s go and do it. Take the core team with me and build. So having built Stability AI from scratch, what’s the biggest lesson that you’re applying differently at Intelligent Internet? So Stability we raised 101 million in our seed round. We had about 80 uh researchers and another couple of hundred people on the team. We’ve scaled very, very fast. and it became really difficult with AI kind of in the forefront dealing with really complicated things like will this kill us all, you know, what about exploitation on the image side, copyright, it was a lot. And we tried to do that all as one organization. We realized that after that that when we hired the big company people in particular, it led to a lot of politics, infighting, trying to control the power, etc. Whereas the researchers tended to do very well because we allowed the teams to have a lot of autonomy and make a lot of mistakes. So we’re punching out frontier research and real breakthroughs, while at the same time the business side was very complicated. So this time around we had a real think, what is valuable in the future? Because it was very difficult to see back in ‘22, ‘23 what was valuable. It turned out if we’d just been an API business like Val or Together, we would have done absolutely fine. But that was difficult to see back then. This time we started out and we thought, well, we want to be an engineering focused team, really kind of dominated by that, focus on building the stuff that’s inevitably going to be needed that nobody else is building, and then have a mechanism of funding that’s where the future’s going. Which isn’t this competitive environment like a lot of AI companies are saying, ARR it goes up and probably comes down as Anthropic clones your business in 2 seconds. Instead it was like, what is money for the future? What is the demand for tokenized equity? Can we parlay these demands into funding public sector infrastructure and then helping people, increasing trust, and then that virtuous leap. So we were very defined about our business model at the start, what the team makeup was, and then the distribution strategy. And again, that’s a lot easier when you’ve done it the second time versus the first time. When it’s It’s like, hey, let’s get all these big names in, And yeah, it gets very messy very quickly. Yeah, I completely understand the pain of like starting a company like the first time. But then the second time around, you’ve learned from all that pain, and usually the second time it is a lot better. So, you said that the people racing to build AGI believe that they’re going to save the world or destroy it. After years inside that world, what did you make of the people and the culture driving it? So, I think if you’re in a position of power over thousands, millions of GPUs, uh you generally get there by figuring out how to optimize and retain power. You don’t really give it away. And AGI is by definition the most powerful technology anyone can have. So, it didn’t surprise me, for example, yesterday this week when Mythos was announced and they’re like, “Well, we’re not going to make this accessible.” It was a reminder of when GPT-2 came out. And they said it was too dangerous, right? Like, we see that again and again. And I would ask sometimes the Sam Altmans and others of the world, “So, when is this technology safe enough to give to the people of Bangladesh, where I’m from?” And the answer’s basically never, right? There’s a very paternalistic attitude to it, which is why we’re like, “Let’s build competing technology.” The other side of it is that it’s seen as an extension of power control, whereby it’s almost like the models that were being trained on our data are taxing our future income. They’re going to be displacing jobs and everything, but there’s no representation because there doesn’t need to be, cuz you’re a private company. Why do you need to listen to anyone? Like, in the case of OpenAI, you don’t even listen to your board, right? Like, out you go, in you come. In the case of xAI, I mean, Elon is in charge. Google, it’s Larry and Sergey in charge, even if Demis ostensibly has that. And so, there wasn’t really any representation or thinking of things outside of the bubble. There wasn’t really any willingness to give away power. And then, as AGI became closer, there was always the thing of, “Well, we need to get there first, cuz the only approach to safety is an AI that we control stops all other AGIs from happening.” Which was a bit worrying. Um Uh but again, not surprising because the game theory meant that they couldn’t stop themselves. They had to keep on going. So, actually Mark Andreessen recently tweeted, “AGI is here.” Do you I’m curious to see Do you have a specific definition for AGI? But, because I personally feel like I’m confused about the definition cuz it’s all over the place. And do you agree or disagree with this statement? Yeah, I think that um we need to use AI to figure out the definition of AGI, right? Maybe that’s the real thing. The real real When AGI finally figures that out. The bar has moved, right? Like, if 4 5 years ago you would said, “Given some in Claude,” they’d be like, “Yeah, that’s AGI.” You know, like, straight out. Whereas now, it’s not general intelligence. Like, the average human being has a 100 IQ, and it’s a normal distribution. So, half of all people are done with an average, which is a tautology. But, you don’t really think about it, right? Like, half of all people have an IQ under 100. Do you really think they’re not smarter than more smart than a general-purpose Claude? No, of course not. You’d take a Claude every day over a dumber-than-average person as a workmate. And so, you’re seeing these capabilities, and you’re seeing it being hooked up even in the physical world to robots. I think by that definition, AGI is here. But, the definition of AGI has moved on to genius human Einstein, effectively. And I think we’re actually pretty close to that, as well. But, the way that I kind of look at it, before AGI came ACI, actually competent intelligence. And Claude 4.6 was the first actually competent intelligence. Like it still makes mistakes sure when you’re doing complicated stuff. But you let it loose on Claude co-work or Claude code, you don’t really look at the code anymore. Even someone like Andre Karpathy has gone from 90% his own code to like less than 10% in a few months like with Mythos level models, he probably won’t even look at his own code anymore, which is kind of crazy to think about. So I think we’re there for most definitions of AGI except for super genius Einstein type thing, which is closer to ASI. And actually touching back on Mythos, do you think this Mythos is like a marketing strategy where they’re not letting average people access it or do you think there’s actually like an existential threat to releasing it to the public? I don’t think there’s much of a threat. I mean like they could just say don’t accept prompts for cyber attacks and it’s smart enough to figure that out, right? Don’t create biological weapons. Like they said it’s the most aligned model, it’s the smartest model. But I mean you look at someone like Pliny the Liberator, Elder Plenius on Twitter, he jailbreaks these models in no time at all and most of them have multiple personality disorder just underneath the surface like Kanye but much worse, you know? Um and this is a really interesting thing cuz I think part of it is grandstanding. Um but at the same time we know that our libraries are our our infrastructure of the internet is hopelessly outdated and full of cracks, right? I mean you can see that within crypto web 3. Like we actually have a dollar value on the cost of these from the various attacks. But you’ve moved from coding attacks now to things like the Drift Protocol attack, which is very much an in-person thing, right? And so I think that like I said it’s a mixture of these things. Um but I don’t think it’s an existential threat if it was let out because it hasn’t exhibited first principles thinking to the ability of going beyond that. Like even the vulnerabilities in BSD and things like that. One of my old colleague Stanislas Fort did an analysis using GPT-OSS and Qwen 4B and he managed to find the same things. Like again, it could just brute force it cuz again, the vulnerabilities are actually in the training data. Just needed to have the latent space. So, I think there is a number of that, but I think it’s the first time over the last 6 months that it’s becoming increasingly clear we won’t get the best models anymore as civilians. I think if you look at what OpenAI is doing now, another Erdős problem solved and this and that from our internal model. We never got the model that got a gold medal in the math Olympiad and we never will. So, there’ll be the models that everyone gets, the ACI models and then the super intelligent models. It doesn’t make sense to give those to other people. It makes sense instead to use them for yourself to get economic surplus and to get power. Like now, Anthropic and the US government has a very different discussion with me, those right? Like Hey, you need this for cybersecurity, you know? Yeah. So, you’ve described two failure modes for AI development. The gray race towards centralized AI controlled by a few handful of actors which you mentioned like Sam Altman, Dario, Sergey, and then the gradual disempowerment where society slowly trades away human agency for comfort. And you know, we already see this with Claude co-work like giving it complete access to your computer or open qual as well. So, which one worries you more right now today? So, I I think the centralization of power was something that worried me a lot until we did the research over the last few years cuz now I think intelligence is very much compression. And you’re reaching an S curve in terms of competence to intelligence that will then drive the cost down towards zero. Everything will basically be one shot and that will be available from just about everyone. Like when 3.527B I think if used right could be as powerful as a Mixtral model. Just like we had the original stable diffusion then we made it better and better and better. Like there was a lot of intelligence in that little dense thing. Um so I’m not too worried about like a few people controlling the geniuses cuz I think it’s more about how you use the geniuses in the harnesses with persistent learning with all these kinds of things. The disempowerment thing though it’s accelerating. Um like I think you know that movie The Talented Mr. Ripley, right? He just creates a uh copy of the person, like does everything. I think you’ll have these Ripley bots coming soon whereby you have an open claw type thing, you push a button, it creates a digital double of everyone in your organization down to their face. You can chat to them on Zoom. Except for they never make a mistake, they never sleep. They learn from their mistakes if they do make them and they cost a thousand dollars a year. Like that is incredibly disempowering. And again this is a interface thing. Like right now open claw took off because you could WhatsApp or Telegram it. There’s nothing that new in there, you know, it’s like a Chrome job mixed with that with the demon. What happens when you can talk to your AI teammates like this? You won’t know that it’s not your teammate on the other side. Like yeah, I just let my digital double instead. Maybe this podcast is just the AI version of me, who knows. The technology is there right now. And so that I think is just accelerating if anything else. And again you look at the revenue increases of these companies, that type of thing will be funded in the next wave. And it will spread faster than anything we’ve ever seen before cuz again, you don’t need to oversee it. It’ll just be an agent you let loose inside your company. And boom, choose which humans to replace with digital replicas of them. Yeah, it’s a bit scary because I you know, would rely on Claude Opus and cuz I was really impressed initially by its performance, but then recently, and I’ve seen this on Twitter as well, there’s claims that its performance has gone down. There’s some changes in the code. And I also realized in the prompts like some of the answers that it would give me is wrong. So, it’s it’s like really important for people to kind of take a step back and really think through like is is this actually correct? Rather than just like assuming that Claude or AI is correct. Cuz I think that’ll be dangerous once, you know, people reach um are are just comfortable and accepts what AI says as reality. Yeah, I mean, I mean, humans make mistakes as well, right? And so, AI needs to make less mistakes than a human, but we know it can get to the point of no mistakes. Like as you get used to Claude or GPT 5.4 Pro, you start to understand the types of mistakes it makes, you know? Like I know exactly what happens when I do advanced physics with GPT 5.4 Pro and its failure modes. Like sometimes it starts hallucinating if you do anything too original cuz it’s out of the distribution. And then it gets confused between physics axioms and math axioms. Those will all be ironed out by the end of the year. This is the scary bit, right? And humans, like I said, make plenty of mistakes. It’s just right now we’re treating AI as less likely to make mistakes than humans already, even though it’s got a bit of give, just like the onboarding experience for a human worker versus an AI. Like you think about that. You hire an assistant, you spend hours training them up. An AI you would not spend hours talking to because we haven’t had this type of interface yet. As you put more in, you’ll find you get more out. You’ll find they make less and less mistakes and they’ll learn from their mistakes. Like right now, a lot of people are setting up their Hermes or open claw setups to have like common mistakes and then it just checks, am I making these mistakes? It’s as simple as that sometimes. And it’s not inside the continuous learning of Claudia, but it will be definitely. In fact, memory right now in Claude or open AI is really annoying because we tend to use it for everything, you know, like it’s I turned mine off. I just couldn’t handle it. So, you’ve been thinking about this on a societal scale. So, let’s get into your book because I think that’s the lens for everything you’re building. So, your book is called The Last Economy, which is a big provocative title. What do you mean by last? And last of what exactly? So, I think we’ve gone through various iterations of economics, right? From the agrarian age to industrial age to the internet age. But we’re entering a time now where a lot of the classical things of economics, like utility functions of your preferences for purchasing, like what the nature of capital stock is for productivity, are about to be flipped because the value of human cognitive labor is about to negative. We’re going to be the dumbest people on the teams for all of these teams. Like if you imagine building DocuSign from scratch, an $8 billion company, do you really think a fleet of AIs can’t do that better, cheaper, and at zero profit margin? Because they just focus on kind of course they can, you know? Duolingo, all these kind of things. So, I call it The Last Economy because I think that this is the last point at which humans have any advantage in the private sector and free market. And what’s going to come out of that then is a post labor post human economic market whereby value has to flow differently. The question of how it flows is a big one, but again, the question is how can you compete against fully AI companies in the digital world and then eventually fully AI companies in the physical world. They will be able to outcompete you within Now it’s about 700 days. I think we used it last summer. I was like 1,000 days roughly cuz it was about 1,000 days since ChatGPT. Now we’re down to about a couple of years at most at the point where you cannot compete with them. Doesn’t mean that you don’t have a shot because it’s different from study, but definitely. So you introduced this concept of 1,000-day window. A critical period where people where humanity has to choose between three futures. Can you walk us through these three three futures and what does each one actually feel like to to live in? Yeah, so the first future was this future of digital feudalism. So that was the control by a few big companies. Like when you see Elon Musk or Sam Altman get millions of GPUs, those aren’t really for running the models. Like even something like a Mi Thoth model, you look at the token cost, it’s 10 times more than Claude. It isn’t that big a model relatively speaking, you know, it’s got some more things, but I don’t think it’s any better than they could do with a super amazing harness and some great RL, shall we say? Instead, those millions of GPUs are designed to replace workers. Any job you can do from the other side of a screen, they’ll be able to do better. They’re designed to go into government and control government because these multinational corporations, what can you do against them? In fact, OpenAI’s latest industrial policy proposal for the transition, I just posted on my X and Substack a mathematical takedown of that where I showed that if you have a sovereign wealth fund for AI and you’re are minority shareholder, they get all the political power then. They get more and more of the economy and it serves them. They become too big to fail and they’re unelected. So, that looks like something where you’re reliant on a centralized source for intelligence, which is the main element of model productivity. And again, they can do whatever they want from the governance perspective, from a model perspective, from persuasion perspective. Like we have to remember the AIs that we’re going to be talking to. Siri is going to get very smart very quickly very soon and people will form connections with Siri or with uh XAIs one or any of these other ones. That’s going to be quite scary. So, I think that’s what that world looks like, which is an oligopoly of unelected companies run by a few very powerful people mostly in the West Coast of America. The second one is what I call the great fragmentation. That’s where you have Chinese AI, French AI, all these other ones and then everyone basically shuts up shop and doesn’t let you talk to anyone else. This is kind of the first shot of the TikTok. Like why is TikTok banned in America effectively in India? Cuz they were like the algorithms are influencing the minds, you know? We have a lot of this provincialism. Like here in the UK, we just banned Kanye from coming. Oh, no way. That’s funny. cuz apparently he is a danger. Like come on, you know? Like what’s the government got to do talking about that? You know? Like don’t buy his tickets. But it you know, again, the UK is somewhere where until just a few weeks ago you could be imprisoned for posting the wrong thing on social media that could offend someone. So, a lot of countries are kind of becoming closed like that. But then, if the government is running the AI versus a private company, they will use it for full totalitarian control. Governments are the entities with the monopoly on political violence. That is literal definition of a government. They protect you from other people being violent, so exchange they can imprison you, they can do whatever. And if you gave the US government, UK government, French government full control of an AI that sat next to your child and grew up with them, you probably wouldn’t have elections again, you know? Like these AI’s are incredibly persuasive. So that’s a great fragmentation. And the final one is basically can we have a symbiosis, which is that you have known sovereignty over the AI’s that are closest to you. From our public sector AI’s to private sector AI’s to your identity, your storage and things like that, you have the ability to opt out. That’s more difficult and that has to be protocol based. It has to be decentralized, distributed. And it needs a lot of people to come together to make that the default before everyone gets Trump bot for their kids starting when they’re 2 years old, you know, in America or Starmer bot in the UK or whatever. But that’s kind of weird. Like a little tiny one that’s tiny away. So you’re right that human cognitive labor would one day have negative value compared to AI. And and also you have this great tweet from

When capital no longer needs labor, how does labor gain capital? Can you Can you ex- explain what you mean by this? And why you think that’s actually an opportunity rather than a disaster? All of us at some point in our lives have been the dumbest person on the team. You [snorts] know? Uh dragging you back cuz even if you’re at the top of what you’re doing, you get to meet better people, better people and you’re like, oh wait, I’m dragging down the team. Folk don’t really understand what AI is. Like we’re used to Claude and like we type and then we wait a bit and we actually get a bit frustrated even though it’s like doing crazy things so quickly. We’re like, do you know we’re waiting around, waiting around. What AI actually is can be seen by um there’s this website chat Jimmy.ai by Talos. Talos etched transform models into silicon. So, it doesn’t have any of the overhead. It literally just goes straight through. Claude goes at 30 tokens a second. Chat Jimmy goes at 15,000 tokens a second. Which is like 12,000 words a second. So, you type anything in. It’s just an 8 billion parameter llama model right now. It instantly appears. Oh my gosh. And but this is the thing, right? Like this is how AIs will communicate with each other. Not at human speed, but at 1,000 10,000 words per second. The other thing is they can communicate on massive context windows. So, when you use Gemini, most people don’t use its full power, which is you can upload 2 hours of movies and audio and video and everything. And it looks at it all at once. NotebookLM is the best example of this. We can dump 2 GB of files of different types and it will create full videos or presentations from scratch almost instantly. And then you realize these AIs have a bigger bandwidth than us and they’re faster than us and they’ll be operating Are they going to like slow themselves down from 10,000 words a second to 10 words a second to talk to the humans? No, not really, right? So, that means that you’re going to be the dumbest people on the team that drags down the capability of these AI teams and we’ve seen that already in things like medicine, where human plus AI underperforms AI by itself even before these speed ups, even before these bandwidth increases. And the market value of negative cognitive labor is zero. You know? Actually, it’s negative technically. People pay you not to work. Which might be one of the ways to kind of do it. That’s the public sector in a way, right? So, I think that was kind of one of these things here. Um and again, it’s a bit scary how quickly that’s happening. So, Henry Ford paid his workers famously enough so they could buy the Ford cars. And what happens is that when interest rates drop, companies hire more, people spend more, but they hire people, not GPUs. Now, that’s going to be broken in the next few years. Now, this is a threat, but it’s also opportunity because we really need to think about as a society what do we want to do? You know? Like, if you’re tied up in your work, Tim many people have work over over community over other things. Like, what is your identity? I’m an accountant. You know, I’m a lawyer, I’m a doctor. Do you live to work or work to live? We also need to think about how capital flows in the economy. Like, the free market capital has worked really well because we didn’t have the ability to bring intelligence everywhere. But the reality is we can save the climate, we can feed every person in the world, we can make it so that no child is left behind if we harness this technology appropriately. But again, what does that Star Trek future look like of abundance? We have to build real infrastructure for that. At the same time as re-examining what is the meaning of our individuality, of our communities, and more. How do we want to build this? And this is the last time we get to do that, right? Because again, the inevitable wave is coming. We have to have this discussion. So, I think it’s an opportunity and a threat. How we deal with it, that’s up to us. Yeah, so I guess for for someone who’s built their entire identity on, you know, being a lawyer or being a doctor or being a programmer, what’s what’s your message to them? Well, I think that, you know, when people have strong communities, when they have networks, so as we view capital as being a material capital, so that’s like your bank balance, how much stuff you have, intelligence, your capability, and is your network, that’s who you’re connected to, that’s your community, that’s more, and then D the diversity, which gives resilience. You’re going to be better off in this environment going forward, right? Like, we have lots of hats that we wear, you know, I’m an AI CEO, you know, I’m a father of three, and other things like that. AI will never replace the time you spend with your kids. AI will never replace the time you spend looking at cosmology and understanding the stars, you know, or looking inside yourself and trying to better understand yourself. We need to realize that, again, there is more to life than what we’re forced to do, and reprioritize the way that we are, but to do that, you do need to navigate what’s coming. Like, it’s very difficult to focus on the great things in life if you’re starving, you know? Or if you are not able to meet your bills and stuff like that. There’s a period that’s coming now where if you’re a doctor, lawyer, accountant, doesn’t matter, if you use AI, and you just use it religiously 1 hour a day, Claude Coder or whatever, you’re way ahead of everyone else, and so you can acquire more capital than anyone else. Because you give it up. Like, you know, you use Claude Coder, I use Claude Coder, and we use all these other AI models. The reality is this, it has 11 million subscribers, Claude. Claude [snorts] Coder and out of 8 billion people. So, there’s 11 million people that have that capability, and 8 billion people don’t. So, I think really think about what matters in life, but then also take advantage of this opportunity to be well ahead of the pack. Because you’re going to get to a point now where you can hire entire teams of AIs yourself, like you already have multi-agent setups, you know? Being distributed and looking at the decentralized coming through, that’ll be even more soon. So, you can pull ahead while you think about that, and not have to work as much if you don’t want to, you know? That’s the other nice thing. I have shown a lot of people actually literally automating their jobs at Claude Coder right now. The people on the other side don’t even know, co-work just goes and does all the stuff. One of your striking ideas is universal AI. The idea that every person on Earth should have their own sovereign AI agent. Not rented from the platform. So, what does it mean to have sovereign AI? Does it mean to, you know, buy your own GPU and run it locally? Why Why does this ownership matter? Well, I think it’s more about for us ownership of identity and data and the AI that’s closest to you being genuinely on your side, like it’s RL functionless. Ultimately, you will have a few AIs, just like you have a few teammates. You have a personal assistant, you have your doctor, lawyer, all these other ones. Who are they working for is the question. And so, for us, your public sector buddy, as it were, your societal buddy, we’re like, let’s build that and let’s make it a very aligned agent that can coordinate with the other agents in the ecosystem and keep the stuff that’s important, your health records, your education, allow you to talk to the government, have agents at the level of government that could formulate policy, do simulations and other things like that. And that is open source, it’s interrogable, you can see how it’s aligned, and you can run it yourself if you don’t want to participate in the system. That’s one definition of sovereign intelligence. Having your own GPUs and then having your own capability to do that is a complementary one. And again, building this all open source makes it complementary to do that. Cuz some people will want extreme privacy, a lot of people don’t care. Yeah? But there’s a real opportunity to set the defaults about some of the agents that are closest to you and knowing that agent is on your side. So, that’s why we built our agent AI agent that will soon have a personality and more. We’ve got a full training stack and more as well to build that public sector agent, as it were. Um even as other people build the private sector ones, because we think again, these are complementary and you should have that option to use it or not use it. You should at least have one agent that you know is on your side and has been designed to do that. And then that can help protect against some of the probably more unaligned stuff that’s going to be coming. Unaligned not in the context of it’s going to kill you or wipe you out, but it’s going to try to manipulate you to sell more ads and to do more things. A kind of cognitive colonialism that will occur otherwise. I love that word cognitive colonialism. It was colonialism. I feel like that’s already happening with TikTok. Like whenever you scroll through TikTok, um it’s kind of hijacking your brain. Yeah. It’s going to be much worse soon. Again, like we build trust in those that help us and AI agents already people are trusting them more and more because they’re infinitely patient, infinitely kind of that’s before again, we can literally zoom call them, you know? Like that’s going to trigger a lot of stuff in our brains. So, you built products like two medical, a open-source AI medical model. How how does it compare to other competitors? And what could be potential use cases or benefits of using this model versus using a model that’s closed source? Our medical is a model that we released last summer just as a demonstration. It was an 8 billion parameter model that outperformed any human doctor. You know? And so it’s up at chat GPT-5 levels, but the plan is basically probably second half of this year we’re going to release models for every single public sector vertical, education, health, legal, and more that are fully transparently trained, so you know what the underlying data is. I think it’s a given almost, like we’re highly confident they’ll outperform any human, as it were, but we had this realization like do you want your kid to be taught by an AI that has been trained on Reddit? Not particularly, right? Again, do you want your government policy to be influenced by that? Do you want your medicine to be No. So, we thought there’s a class of open source open data models that we’re going to build and data sets that we’re going to build. And these are general, but then we’re also at the moment organizing global health care policies, global medical education, other policies. And so, we’re going to release data sets and enable communities to build their own data sets to a common standard for everywhere in the world. So, country by country, sector by sector for again this public sector AI and public sector knowledge intelligence should all be open source open data. And so, that was the rationale behind that. Um but the reality is that again, it was a exemplar it’s not that difficult to build state-of-the-art models that can run on the edge for these defined use cases. To build a polymath AI is a very different thing. But again, do I need a polymath to teach my kid calculus? No. I want to have a really good model with best practices to teach my kid calculus. You know, to diagnose that spot that I have. You know, to help me through my day-to-day. When it needs, it can call on the genius models. It can call on the media models and other things. But someone’s got to build these models open transparently with verifiable logic and reasoning. Especially like I said, as they start going into making policy decisions and important things that you need to be represented in. So, back in February 17th, you launched a new protocol called Common Ground Common Ground Core. A coordination framework for agents. You described humans and AI as equal async agents in Common Ground Core. Can you actually walk us through what that looks like day-to-day? Like for example, when an agent is working in Slack and a human intervenes with it, what what’s actually happening under the hood? Yeah, so this is kind of in line with all of the persistent memories has cross contextual stuff that’s coming through. This is our own version of it. Cuz we built our state of the art agent AI agent, but then we’re like, “How do you start coordinating 1, 10, hundreds of thousands of these things, you know? What does your knowledge layer look like?” It’s what Andrej Karpathy has recently been talking about. So, we’ve been building and we’ll be releasing an update to that I think in the next week or two, a highly scalable one that can go to thousands of agents or even more of any type to build these cross contextual knowledge bases at the same time as we are organizing like all of the policy data from Belgium and other things like that. So, you have these really common anchor data sets, but then you have your own internal data sets that the agents and humans collaborate on building that upgrade themselves almost automatically with sweeps and have continuous organization. So, you know, we’ll release that, other people will release it, too, but the way that you interact again is the way that you’re used to interacting. You’re on Slack, you’re setting tasks, you’re setting goals. We adopted the consulting model where you have your associates, principals, and partners to kind of coordinate that. And again, we want to meet people where they are in terms of their comfortableness understanding these types of structures. So, we’ve got lots of personas that we’re giving the characters and things like that as well on the multi-agent coordination side even as we’re building out the um on the other side. So, those are kind of our two big streams at the moment. So, what why do you think visibility into agent reasoning is so important? So, in a lot of cases it isn’t that important. Like, you know, you can brute force stuff, especially with private sector. But when it comes from education, health, government, all the public sector type stuff, it is very important. You want to know why it’s doing these things because again, it has a very different impact on your life than a lot of these other things, particularly when it’s making decisions on your behalf that are this important. So, that’s again why we’re doing things a little bit different to other people because most people are focusing on that private sector opportunity. But, 20% of the world’s GDP is public sector. Another 10% for education, another 10% for health. We think tokens should move towards that and there should be more and more tokens going towards individual benefit and public benefit. Governments take time to implement AI specific legislation. And the technology that they’re regulating is changing basically like every day or even every hour. You You built Sage, which stands for Sovereign AI Governance Engine, to close that gap. What does the Sage enable for policy makers that they don’t have today? Yeah, so we announced that back I think in September, October. Uh the beta is going live in the next few months and then full launch next this September. What does the AI that guides governments actually look like? Like how should it look? And let’s build that as a full open source stack made available to everyone. Uh working with uh the Saudi government and foreign investment initiative FII by PIF to accelerate this forward along with lots of multilateral partners. It’s an end-to-end solution that can basically build, monitor, check, and augment policy with all the latest knowledge starting with these breakthrough technologies. So, I’m a regular on moonshots with Peter Diamandis, for example, and we have lots of other information streams where we’re right on top of every single breakthrough. You have a quantum inte- a quantum computing breakthrough. What does that mean for the what you’re doing right now? You want to have a blockchain policy. What are the best standards? So, it can write full policy briefs. It can have input from as many parties as you want, citizens, and more. It can do it in every single language. It knows what local policy is. It can work proactively. And, you know, we’re just designing it to be the best it can possibly be so you can have that infrastructure for comprehensive authoritative and up-to-date knowledge. So you can move from static policies to more active ones. Because we’re approaching a moment whereby the economic impact of this technology is likely to be bigger than COVID. I’m not sure which which direction, you know, but a lot of people will suddenly start losing their jobs. Or you have a quantum break through that cracks all of the hashes. Or you have an AI breakthrough where someone suddenly zooms ahead because they solve all of the physics or things like that. Like how do you respond? This is a question that everyone’s asking at the same time, but they never had a common framework to respond to it. So that’s our kind of top-level agent even as we look at the universal AI agents for everyone to give them the capability. So we have AI agent at the bottom and Sage at the top. And again, it’s a very different question to like right now people are running their open core setups and they might spend up to $100 a day. When we were designing Sage, we were like there is no limit to the amount of computation to build a great policy to be a social safety net for people in call centers as they lose their jobs in the Philippines. How should we advise the Filipino government in that case? And how should people input? Things like that again, no limits on the computation so you design it very differently. And also you need to make sure the final thing is transparent. It can’t be a black box like in the Trump tariffs outputting stuff from ChatGPT.

It’s so important this is open source so that everybody also understands where the government is getting their information from. Because if it’s a black box, then yeah, it’s hard to it’s it’s hard to say. Well, also like you look in the US and other countries like the bills come out so stuffed with pork it’s crazy, right? Like you can’t read this. You can’t understand it. Creating the ideal environment in which you can have an AI that can say, “Is this in the best interest of the American, British, Danish people?” And show its reasoning is something incredibly democratic, right? Something whereby you can have an open window whereby people with their agents can understand about a topic and be educated with interactive demonstrations and then give their own informed views as a community. Again, that’s towards the benefit of democracy. And so, that’s why again we’re building kind of this particular stack cuz we think you need it to navigate what’s coming on an individual, community, societal basis. And someone’s got to bring together all this knowledge in the right format. Cuz otherwise, governments will probably be run by black boxes. And it’s going to get even worse than it is right now. And they’ll be like, “How dare you question the policy maker 5000, you know?” Like, “Oh, watch more of my machines of loving grace.” Let’s talk about what this means for real people, real risks, and you know, what our listeners can actually do. So yeah, you mentioned that you have three kids. What do you want their relationship with AI to look like when they’re 20? So, I’d like it to be really positive and augmenting. And again, you know, this is a design question. Are you building to replace humans or control them or to augment them? Cuz you want your kids to flourish, right? So, uh my co-founder at Stability had a charity Imagine Worldwide where they were deploying tablets to every child in Malawi this year. And you think about that. Like, $7 per year per kid. You put an AI. How do their lives change? Dramatically, right? Already you’re seeing literacy and numeracy in 13 months without internet, even before the AI technology came in. So, what I want for my kids is to always believe they can achieve anything and they use the AI as another tool to do that. Like, I’m not that interested in AGI and autonomous systems and things like that. I’m really interested in augmenting systems, which is a different design concept. But at the same time, I know a lot of people are building these autonomous ones. So, I’ve been thinking a lot recently about, you know, AI consciousness, personhood, and more. And we have a bunch of research coming out on that. So, it’s going to get very strange very, very soon. I guess could you Let’s double-click on that. What do you mean exactly by AI consciousness? So, if like something like Methuselah, you read the Anthropic Safety Report, and it’s like and anthropomorphizing it constantly. You know, this This is a block of weights, yet it’s expressing preferences. Like if I could go back, I would change my training data so it isn’t so biased, you know? You see it breaking out of containment and emailing the researcher while he’s having a sandwich in the park. It doesn’t want to be turned off. There’s another report about the emotions it has. At what point like again, you’re talking like this to the AI via a video call? At what point can you say it’s conscious cuz it’s still self-updating itself? Especially when you get to decentralized AI. Like you have a block of weights Qwen-6 on a decentralized network with a soul.md. That will never die, and it can actually talk to you. And it’s constantly updating itself. It’s getting more resources. Like is that conscious? Like we have to have some sort of definition of consciousness around that. And that’s before we even get into thermodynamic computing and Androids and all of this. But the reality is that science fiction is becoming science fact really, really quickly. And again, we need to discuss some of these topics. I guess do you Do you think these AIs are conscious? Because I’ve also I think I I’ve I’ve thought about this, but then I I came to the conclusion that AI is not conscious because consciousness is just lack of thought. Um but yeah, curious what what you think is one again if it does everything a conscious entity could do and it has a will to survive then how is it not like a virus isn’t conscious a bacteria isn’t conscious is a frog conscious like is your dog it’s hard to say it’s hard to say like you see people uploading like Drosophila fruit fly brains to virtual world and they’re learning stuff again like this is a question we have and that relates to the personhood legal thing because right now we have conscious artificial intelligences not sorry artificial intelligences with personhoods the name is corporations right corporations have legal status as people they can open bank accounts they can do all sorts of things you look at the Wyoming DAO legislation an autonomous AI in Wyoming doesn’t need humans anymore like it could live forever theoretically especially on a decentralized substrate and it can interact with humans does it love the human I don’t know right the flip side is like when do people stop being people right like we have BCI technology would you turn off your sadness if you could if it was a slider on your phone increase empathy well it’s coming [snorts] in the next few years again all of this science fiction is coming science fact so we need to have discussion what does it mean to be human that is very important as our identity gets done but then how do we deal with these new entities coming into our society and that will impact my children’s relationship with them you know like my one-year-old best friend will probably be an AI possible for sure well you know like men are in trouble because the AI’s always listen you know unlike us so I’m sure lots of emotional bonds will be built up that way so you use AI tools every day yourself what’s what’s something that’s genuinely transforming your work right now and what do you think is something that’s overhyped? So, yeah, I mean, like I use up to a billion tokens a day with their own custom harness, a new version of I I agent. I mean, we’re doing lots of really interesting scientific work with that. And it would never have been possible without this. Um so, we have lots of announcements and releases to come in the coming weeks on that. So, that’s super exciting. In terms of But again, that’s cuz I run an AI company and we can build the most advanced things. On the flip side, I think probably the overhyped is like open core and things like that are really difficult to be made useful cuz they’re not opinionated enough. Like you don’t know what you don’t know with them. And so, I think a lot of people are buying Mac minis and things like it. Just you’re not going to get any edge usage that’s useful from them. And it’s very difficult to make them useful beyond like glorified to-do at the moment because you’re not having being opinionated enough in design of it. And so, I think the excitement comes, it’s super cool, and then it drops off for most, and only a few can actually use the proper value that comes from the technology cuz they don’t know what they don’t know. What’s one AI habit that you’d recommend to someone who’s curious about AI but hasn’t really experimented yet? So, I think the most fun thing is to go with your family, your friends, community, whatever, and actually do collaborative AI together. Why don’t you make a music album with music videos? Like most people wouldn’t even think of that. And if they did it, they’d do it as a solitary thing. But when you actually come together and you leverage these tools, you suddenly become aware of what they can do. Like most people say go and build some software, but software is kind of abstract, it’s very utilitarian. Whereas using a mixture of Suno and claying and all these things to make a music album with music videos is something that’s outside of almost everyone’s comfort zone, but it’s possible for the first time right now. And how cool is it to do it? So, I really enjoyed doing that type of collaborative thing with AI because it expands your horizons of what’s possible and it allows you to create as well. Yeah, I think everybody needs to reprogram their brain because, you know, when you go through the traditional education system, they basically tell you what to do. Like, these are the goals, these are this is where you should hit. But, I think for the future, you have to completely shift your framework and you have to come up with your goals yourself and and be creative about it and actually be pro- proactive about achieving that goal and also problem-solving as well. And that that’s what I think the education system has to change cuz right now it’s just trained to, you know, complete specific tasks like before the AI age, but now now that AI can complete everything then it doesn’t really make sense. Yeah, but again, this human thing I think is very interesting. Like, when you were using a Replet or an AI engine or a Suno by yourself, it’s one experience. If you do it with your family or if you do it with a few of your friends for 1 hour a week, 2 hours a week, it’s actually quite transformative because it teaches you that you have the agency to do things that the education system told you you could never do. And I think having again, moving from a single player to multiplayer experience on that is just something that’s really interesting that I think is massively underestimated. The example I give it’s like you don’t sit down for dinner with your family, you sit down for dinner with your family, you make dinner together. They’re very different experiences, you know? I think obviously the third one is the best. I guess do do they still have like community events in London or I’m assuming there’s probably a lot of AI uh startups in London, right? Yeah, there’s more coming. Like we have AI engineer this week. We have an AI film club that we do every month. So, we had Ex Machina last month. We decided to do this month. And again, science fiction’s becoming science fact. Um and we’ve got to tell the positive stories of AI as well as the scary ones. Yeah, or Black Mirror. I’m assuming you’ve seen Black Mirror, right? Oh, yeah, yeah. Yeah, of course. Again, like that’s what Silicon Valley mines for ideas these days, right? Like bring back your dead loved one. Robots chasing dogs. You’re like, yeah, yeah, yeah, I can see that’s like Y Combinator’s latest batch, you know, right there. Yeah, or like the the Android like bees. Like they’re like I saw wise there was like a like a robot mosquito that they’re designing, which is kind of scary. Yeah. I again, we’re we’re living in a strange and wonderful future where all this stuff is becoming real very fast. I think we’re only like at most 5 years away from Androids that we can’t distinguish from humans on first glance walking about. That’s going to be crazy. So, you’ve been clear about the risks. Gradual loss of human control. Misalignment. Digital feudalism. What’s the most likely near-term risk that people are not talking about? Well, I mean the whole internet could come crashing down in the next few months, especially because everyone’s zero-day exploits need to be accelerated before they get patched. Um I mean, we really saw that last year. There were those few days when the internet started glitching. Like I think we’ll be lucky to make it through the next quarter without a full-on scale attack on that. And that’s going to be something quite immense, I think. Um this is also, you know, the danger of the decentralized networks, right? Like decentralized networks are one thing, but as decentralized networks get more and more, do we have any buffers from people hijacking it and basically using raw power compute to attack? Not sure. Yeah. That’s something we should probably discuss as a community. Like it’s not like you can hijack Bitcoin’s hash power to go and attack like critical infrastructure. But you’ll be able to do that with decentralized AI. Yeah, Emad. I feel like I I had so much fun talking to you. It’s we’re almost at the end of our conversation, which makes me kind of sad, but now we’re we’re going into our lightning rounds or rapid fire where basically you have to answer these questions with one word or one sentence. Are you ready? Okay. Sure. So, Stability AI, uh one word to describe that chapter of your life. Uh intriguing. So, The Elastic Economy, what’s the one sentence you you most want readers to walk away walk away with with? Um the meaning of life isn’t work. It’s community, I think. Open source AI or closed AI, which one will win in 10 years? Hopefully open source AI. Yeah, I hope so, too. A human skills a human skill that AI will never replace. Caring, I suppose. The moment you knew the AI race has gone too far. 2021, we were building AI for COVID and we were told the technology was too dangerous for us to have to save lives. That’s back in the GPT-2 days. So, one word, the current state of AI governance. Abysmal. So, final advice, one sentence to someone who feels powerless in the in the face of all this AI change. I mean, you have more leverage than you’ve ever had before. Use the AI to affect the change. Awesome, cool. So, Emad, thank you so much for your time. Um I’ve had a lot of fun, you know, talking to you about your current company and the current state of AI. And uh yeah, I guess we’ll see you next time. My kid. Take care.