Oxford Genius Ai Will Become Earths Dominant Mind Nick Bostrom

TITLE: Oxford Genius: AI Will Become Earth’s Dominant Mind | Nick Bostrom CHANNEL: This Is The World DATE: 2026-04-25 ---TRANSCRIPT--- Maybe God can do anything that is logically consistent, but simulators would be subject to the physical constraints operating at their level of reality. If they didn’t want us to notice any glitches, I think they would be able to prevent us from doing so. In the long run, I think AIs will be completely cognitively dominant. Nick Bostonramm is one of the most provocative minds of our time. An Oxford philosopher who dares to question whether reality itself is nothing more than a simulation and whether humanity stands on the edge of creating intelligence that could surpass and redefine us.

If we are in a simulation, I think for example, an afterlife is a much likelier prospect. What happens after if if one simulation terminates, we need to make more progress on AI alignment. We need better mechanistic interpretability tools to be able to see what’s going on inside these AI minds and understand it. Why do we find ourselves alive right now in 2026? Is it possible that we’re just Nick, honor having you here. You wrote this. One could equally maintain that if nobody created everyone dies. In fact, most people are already dead. The rest of us are on course of follow within a few short decades. Should we create super AI to prevent it? Yeah, I think one of the most obvious uh gains from machine super intelligence would be the ability to massively accelerate progress in medical science. Um so diseases that maybe we could eventually get around to curing you know in 50 years 100 years I think there would be a good chance of uh those becoming cured very shortly after the development of AIs that can sort of do all the medical thinking and maybe run the experiments and so forth. Um so I think there is some urgency. I mean, if you look around at the condition of the world as it is today, it’s hard to feel very complacent and think, “Ah, well, let’s let’s just take another few centuries to sort this out.” That there’s kind of a lot of desperate need out there that uh um so there’s certainly a cost to delaying. Does LLM plateau before it gets dangerous? Is this jaggedness permanent? Well, I think eventually um even if there remains jaggedness, the part where the jaggedness goes inward will still exceed the disk of what what the human mind is capable of. Um I think there are fundamental reasons to um think that information processing uh in artificial substrate can just vastly outmatch what’s possible in a biological substrate like a human brain. So in the long run I think AIs will be completely cognitively dominant. Um and I sounds like a bad scenario. Well or or a good scenario. It sounds like a scenario in which there will be a fairly profound transformation uh in the world um from the current condition where we humans are responsible um for everything uh to to a condition where we might increasingly rely on uh AI acting on our behalf to take care of most of the instrumental tasks that need solving. It means that our race, human race is going to be dead. Dead. Well, hopefully not. I mean, we humans have superior cognitive capabilities and technologies relative to um say many animals and and uh they are still around um not always treated as well as they should be. But we have one key advantage with respect to super intelligence which is that um we get to build it uh and therefore potentially get to design it and shape it um in accordance with our values. And so the idea is to develop these AIs such that they are actually um harmless but also helpful to humans and can kind of empower us uh to achieve more of what it is that we are trying to accomplish in the world. Um and that would presumably include uh human flourishing. Uh so far from killing us the ideal would be that they would be saving us from all kinds of other sources of mortality to which we are subject in the status quo. Where do you see the biggest challenge for current stage of AI development? Well, there’s the the safety silences is is key in terms of ensuring that the future goes well that we need to make more progress on AI alignment in terms of just getting capabilities up. I think I mean we’re already seeing extremely rapid progress. I think continuous learning would be a significant unlock if that happens um perhaps later this year or next year. Um I think the sample efficiency seems to still be lagging behind humans that if you think about a human child who goes through school and maybe university if you look at the number of words that they have read by the time they become a sort of say a fully competent adult um it’s a small fraction of the number of words that an LLM has read during its pre-training phase. Um so we we seem to have this ability from a relatively small corpus of data to develop the full panoply of normal human adult capabilities whereas current AIS seem to need more data to achieve comparable levels of performance. So there might be some unlock there as well if one could figure out sort of how to make these AIs learn more efficiently from a given set of um data points. Um then of course we have sort of physical manipulation where uh humans are are are quite dextrous uh like I think we have a long biological heritage that has endowed us with kind of maybe specialized neural circuitry for motor control out of distribution generalization. This is kind of when you’re able to think uh reasonably and creatively outside a domain that you have been taught. Um where maybe humans still have some advantage whereas the AIS excel in situations where you have a lot of training data doing the same kind of task but then transferring that to a completely different context there is some question of how reliable current models are. um that’s probably related to sample efficiency. Um so yeah, there are still some some areas um where current systems are lacking, but if you look at the trajectory, the the the pace of progress, it’s really quite astounding. And now we’re beginning to see AI is feeding back into AI development actually helping accelerate AI research with the coding assistance being heavily deployed in the leading AI labs. Um so that creates this feedback loop right so now each generation of AI says that get smarter can also contribute more to uh the development of the next generation. This is one of the dynamics that can potentially lead to an intelligence explosion. What’s about AI safety? Maybe we don’t need AI safety researchers. Maybe we just need a stock market crash. I don’t really see how a stock market crash um would solve the alignment problem. It might temporarily uh reduce investment flows into the buildout of AI data centers by some modest amount, but then presumably u the stock market would recover and the flows would resume. Um so I think there is no substitute to actually solving these technical problems. Um to get scalable methods for AI alignment. Um we need better mechanistic interpretability tools to be able to see what’s going on inside these AI minds and understand it. Um increasingly we will need to I think leverage the capabilities of these AI systems to assist in the alignment of AI systems. So we could have sort of weaker AI systems that are more trusted supervising and monitoring more capable AI systems. Um already the AI coding tools are of course used by human AI alignment researchers. I know a lot of researchers also from OpenAI who think that alignment is fundamentally unsolved. I don’t agree with that. Um, and I actually haven’t heard anybody arguing for that. I think the question is how difficult it is. Um, in principle they can be very smart minds that that are nice and care about humans. I mean, within the human distribution, for what it’s worth, we see, you know, some human geniuses are nasty and evil, and some are really nice and pro-social and friendly, and there seems to be no necessary connection between some level of cognitive capability and and your motivations, whe whether you care about other people, you could have any combination, right, of stupid, dumb, smart, nasty. Um and um the same I think holds in in the space of digital minds that can be created. Um and I think in fact there would be some super intelligence if we could create it would be a lot nicer than than than humans. You know we we are kind of the the side product of an evolutionary process that didn’t even try uh to make nice humans. It was purely optimizing for the ability to survive and have a lot of children. Often that involves kind of dominating tribal contests and, you know, stabbing your enemies in the back and all kinds of stuff, being jealous. um in the context of AI, we have the opportunity to actually deliberately try to select and steer the process that generates these AIs with a view of trying to make them um ethical or friendly to humans. Um so the chances in principle should be excellent that we could do that but there are a number of ways in which things can go wrong and that one needs to be careful about. Can you walk us through the simulation trillemma? the simulation argument. This was this paper I published back in 2003. Um, shows that at least one of the following three propositions is true. It doesn’t tell us which one, but the three alternatives are one that almost all civilizations at our current stage of technological development um go extinct before they reach technological maturity. So they sort of fail to technologically mature. Um that’s one possibility. Another is that there is a strong convergence among all technologically uh mature civilizations that they all lose interest in in creating um detailed computer simulations uh of of people like their historical predecessors or variations thereof. So they at technological maturity would have these enormously powerful computers. So they they you could sort of do basic um modeling of this you could have planetary sized computers built with nanotechnologies like enormous computational resources and they might spread through the galaxy building these structures and yet they wouldn’t even use a tiny fraction of all of these resources for that purpose of running ancestry simulations. So that’s the second possibility and then the third is that we are living in a computer simulation uh built by some advanced civilization. Um so the simulation argument shows that uh at least one of these is true. It’s an important constraint I think about what you can coherently believe about um the structure of the world and our place within about the creator. In the third possibility, our world would have a crater in the sense of the people having built the computer uh in which we are running uh would in a literal sense have have created um us and and the world. Now that is consistent with there being a kind of more fundamental creator like so like God might have created everything including this advanced civilization that is then building the computer on which we run like just as you know you can you know breed a bunch of ants in in a glass jar like in some sense maybe you created those ants by breeding them but still you know your parents created you and and maybe uh the sort of divine creator created the whole thing so it’s it’s like a logically consistent with that either being the case or not being the case. But functionally, what is the difference between a a creator who speaks the universe into existence through and the simulator who executes code? No. So, it’s not meant to be a metaphor, but in a literal sense, if the third hypothesis is true, um then the world around you, the uh the microphone that that you’re looking at there, you know, your sweater, all of this stuff, um exists as information being processed inside a computer that is built by some advanced civilization somewhere. And it has some power supply and it’s you know generates heat that has to be reflected and um um it it it’s not it’s different from the idea which with which it is sometimes confused that like you know there’s like a kind of perspective where you could look at fundamental physics as in information processing terms. So rather than thinking about stuff like billiard balls bouncing in, maybe we should think of the basic constituents of the universes being bits. Um, now that seems to me more a question of which theoretical framework is most uh elegant for organizing our basic physical postulates. Um but whichever way that goes, it’s a separate question whether our world was literally uh brought into existence through a physical process that involved an advanced civilization creating a computer and then using it to implement us. Regarding the simulation trilmma, don’t you think that such controversial despouse physics to the charge of being unfalsifiable? So if you’re a basian and you think in terms of probabilities, right? Um and I got to sum to one. So anything that is um if there is something that could be evidence for the simulation hypothesis then the absence of that is evidence against the simulation hypothesis. Right? So now we can think are there possible observations we could make that would increase the probability of the simulation hypothesis now we have this simulation argument that at least one of these three is true. One thing that would increase the probability of the simulation hypothesis is evidence against one of the two alternatives. So for example, if we get close ourselves to the level of technological maturity that would allow us to create these ancestor simulations um then that would be strong evidence against the first uh possibility like if if we get through to technical maturity then it’s almost certainly not the case that virtually every civilization at our current stage of development fails to get there. Similarly with respect to the second one, if at that point where we’re still where we reach a technological maturity, we still remain interested in creating ancestor simulations, that would be strong evidence against the second alternative. And so imagine the situation where we have built like we’ve reached technological maturity. We have, you know, created all these massive computers. We are just about to reach in and sort of press the on button that will start um generating ancestry simulations. At that point as we press that button we would have very strong evidence that we are in a simulation. Um now there are other ways like for example if a window popped up in front of you saying you are in a simulation click here for more information that would also be pretty conclusive evidence. Um so clearly there are possible observations we could make that would be evidence for the simulation hypothesis. Therefore, the absence of those um anything that makes it more likely that we will go extinct before reaching technological maturity or any thing we perceive where we sort of converge towards thinking we should never run ancestor simulations at technological maturity. The absence of any windows popping up. All of that would be probabilistic evidence against the simulation hypothesis. If we can’t find glitches, you say the simulation is good. If we find an anomalies, you say it’s evidence. Isn’t an falsibility exactly what we criticize religion for? People criticize religions for all kinds of reasons. Um, no, I think the fact that we are not seeing glitches is some evidence against the simulation hypothesis. It’s just very weak evidence because it would be likely that uh these super intelligences that would be responsible for the vast majority of simulations that would ultimately be run would also have the capability of preventing glitches of a sort that would be detectable by the creatures the observers inside the simulations. Uh, so if they if they didn’t want us to notice any glitches, I think they would be able to prevent us from doing so or or they could even edit out our memories afterwards if necessary. Most philologians say God is constructed created by logic. If both creator and simulator operate within the same rules, what’s the meaningful difference for you? simulators would presumably be subject to a lot more constraints than the laws of logic. Um so maybe God can do anything that is logically consistent. But simulators would be subject to the physical constraints operating at their level of reality. Um they would have to build this computer. They would have to ensure that it has sufficient memory capacity. Uh you know they would need to repair it if it breaks down. There would be a limit to the number of these computations they could perform. They would be more limited and they would also be yeah subject to all kinds of limitations operating at their level of reality. You know maybe maybe they have issues with their their neighbors uh you know that they need to defend their structures. um maybe they um face various natural catastrophes like from our point of view inside the simulation of course we have a little inkling as to uh this uh deeper level of reality. Um we can however recognize that it does seem to increase the space of possible future scenarios for us that there are certain things that at least if you’re a sort of have a naturalistic outlook like suppose somebody who’s not religious they might think well after you die there’s really not much hope of an afterlife because we can see the brain literally rots or if it’s cremated all the molecule scatters so all your memories are erased. Um if we are in a simulation I think for example an afterlife is a much likelier prospect. What does the simulation hypothesis actually give as that theology doesn’t? I guess first of all theology is not just one thing but there are sort of many different theologies that I guess it’s less I mean it does allow you to ask new questions but primarily it allows you to answer certain questions. Uh um I think the main uh reason for being interested in the simulation uh argument is that it’s uh actually um supported by this specific line of reasoning. I mean, we didn’t go into it, but that’s kind of what makes it interesting to me, rather than it just being like, oh, imagine if you know what if this whole thing is a dream. How can we be sure not dreaming? Or maybe there’s like a demon that is deceiving us. And so these kind of skeptical speculations are old. They go back to you know ancient Chinese philosophy and like Decart was like this long history in philosophy right of using this as a sort of inepistemology the study of knowledge as a sort of challenge like prove to me that the external world exists prove to me that you’re not dreaming and then if you’re interested in epistemology it’s a kind of an interesting thing you can philosophize about. Um but the simulation argument starts from the assumption um that everything is as it appears to be. There is this physical world, you know, it has tractors and uh air ventilation systems and computers and um there is technological progress and then thinks through what the consequences of that is which is that you would eventually reach a stage of technological development where you have the ability to run these vast numbers of ancestor simulations. And then it thinks further what it would mean if the world is such that there are all these uh ancestor simulations with brains like ours in them being simulated at a sufficient level of detail to be um generating conscious experiences. Then in in that world most experiences of the kind that we are now having would exist in these simulations, right? rather than being implemented by basement level biological reality. And so then conditional and that being the case, we should think we are probably one of these majority um simulated ones rather than the rare exceptional non- simulated ones given that from the inside you can’t uh tell the difference. And so then then you’re led to this interesting constraint that at least one of those three alternatives is true. um which which kind of narrows down uh the ways the world could be um in a surprising way and that to me is the most um interesting thing about the simulation uh argument. Now, of course, then it does open up a whole bunch of um further questions as to the motives of the simulators. Um what happens after if if one simulation terminates, when will it terminate? Are there conditions that would result in it terminating? What what more broadly is the distribution of these different kinds of simulations that might include people like us in them? Why do we find ourselves alive right now in 2026? This seems to be a kind of peculiar time given that we are sitting just at the threshold to the development of super intelligence. Um most people uh lived a long time ago. Uh maybe they were hunter gatherer farmers in Egypt or something like that, right? And and right now we are sitting here talking about the prospect of this like most important thing in ever human history and we are really close to it. is one of the option that we are puppets deep inside a quantum computer. Let’s first decide the quantum I think I think it wouldn’t be necessary to use quantum computers in order to um perform these kinds of simulations although maybe it would help. Um now what do you mean by puppets that everything is created by simulator? I mean it might be that our behavior is controlled by our brains. uh the brains were created perhaps by the simulators or maybe they didn’t even directly control create the brains but they just created the world in which brains arise through natural simulated processes. Um so there could be different level amounts of indirection that could be used to achieve a specific outcome. So I will ask you the another question. Pascal had his wager. What’s the simulation wager? Why should I try to be interesting to the simulators? I wouldn’t necessarily say that you should. Um it’s not clear that entertainment is the most likely motive um for running a simulation. It’s difficult to know exactly what would make most sense if you sort of conditionalized your actions on the simulation hypothesis. I would say that to a first rough initial approximation probably you should just keep going on as you would have anyway like because even inside the simulation you you still need to perform various actions in order to sort of sustain yourself um for the same reasons as you would in a basement reality. U now on the margin there might be some ways in which our predictions diverge. If you conditionalize on the simulation hypothesis, I alluded to uh afterlife seeming more likely than it would have been if we conditionalize on a naturalistic non- theological outlook and the simulation hypothesis being false because in that scenario like the afterlife doesn’t really u but there might be others to the extent that you think you have speculations about the motives for creating the simulation and that might also uh shift your expectations about what the consequences of different actions would be. Um I think we are fairly ignorant about the sort of psychology and motivations of these simulators. Although one can uh entertain various speculations, I I would be hesitant to um draw any strong inferences from that. You describe science as pulling balls from a giant urn. Most are white or gray. But what happens if we pull a black ball? Well, so this is from the paper I wrote a couple of years ago um called the vulnerable world hypothesis. Um so the earn analogy is a sort of metaphor for our technological uh progress where we invent new technologies. There are sort of a set of possible technologies permitted by our physics and we are pulling out these one after another from this giant earn of possible inventions and um throughout history for the most part all these technologies we have discovered have been beneficial. Um some have been a mixed bag. uh we haven’t yet pulled out one like a black ball in this analogy that is such that it sort of by default destroys the civilization that discovers it. But um we could imagine counterfactually a number of technologies in the past if some parameters had been slightly different could have been black balls. So think the discovery of how to unlock nuclear energy by splitting the atom and then the atom bomb resulting from that. Now it turns out that to to make an atomic explosion you you need some very hard to obtain ingredients you need highlyenriched uranium or plutonium right that these are these are difficult huge industrial facilities tons of energy only really nation states can do this. Um now suppose it had turned out to be different that there were some easy way to make a nuclear weapon like you know maybe you have a microwave oven and some glass and some sand and you bake that in some special way and you get an atom bomb right so now now we know that that’s not physically possible but before we did the relevant nuclear physics how could we have known um so in that counterfactual where it would have been sufficiently easy to create nuclear weapons then maybe that would have been the end of civilization the worst scenario. Well, it would certainly have been a bad scenario if because it would have democratized the power of mass destruction. So, probably you couldn’t have cities if everybody had the ability to destroy a city because in any sufficiently large population of several million people that will always be somebody, right, who is either evil or goes mad or is trying some extortion thing. Um so we could end up with that situation with biotechnology if it turns out to be sort of offense dominant at some level of technology. Um and there could also be more subtle ways in which the world could be vulnerable. Um not by empowering individuals to create great harms but by um technologies that create incentives for powerful actors uh to create great harms like if there is a strong first strike advantage for example. um and some other ways as well that the paper discusses. So it seems like a hypothesis one can entertain that the world is vulnerable meaning that there is at least one black ball in this urn. Um and if it is then you know if we keep making inventions then eventually we will get to that and then be destroyed by default. Um and the paper then discusses what would be required to avoid that if if there is a black ball in the ear like what kinds of structure would the world need to have in order to be able to survive the discovery of such a black ball invention. Um so I mean I think it’s an open question whether the vulnerable world hypothesis is true. Should we stop doing science so openly? Not in general. I think there could be some narrow specific areas where it might make sense. Um, and it’s already the case, for example, with biological weapons research, which is prohibited, but there is a borderline of research that is aimed at developing defenses against biological weapons, which sometimes kind of involves doing similar things as you would be doing if you were actually developing biological weapons, because you need to sort of think about what you’re trying to defend against. I think some of that research probably should be done only in a classified setting and not published. Um, and there could be other narrow specific areas where it just looks like either the research should not be done at all or if it should be done, it should be done in a way where the information can be contained. I I feel that your solution to existential risk sounds like a high-tech panopticon. think is humanity survival worth a world without privacy? Well, so the high-tech panopticon is uh uh introduced as a concept in this paper on the vulnerable world hypothesis. Um so the paper thinks about if it did turn out that there was this black ball like how could civilization be stabilized in that scenario. So then it depends on what precisely this black ball is. But if you look at general solutions um for this class of possible discoveries that would empower individuals to create unacceptable amounts of harm like for example the easy nuke scenario or if there were some easy way to make biological weapons that didn’t require sophisticated lab equipment and so forth. Um then in in if if we are unlucky and the world were like that then it might be that the only way for civilization uh not to be devastated would be if you have this extremely intrusive um surveillance mesh that continuously monitored what everybody was doing. So if you can’t remove the materials that would allow people uh to do this enormous mischief because maybe the materials are ubiquitous um then you would have to sort of check what they were doing all the time so you could intervene if anybody started to build this this mechanism. Um so so that’s and on the other hand you have for a different class of ways the world could be vulnerable you might need something like um a global governance that would be strong enough to solve some of the global coordination problems that can otherwise create arms races between nations or incentives to strike first. Um, of course, these uh stabilization methods themselves would be accompanied by great and obvious risks with surveillance. It would make it easier to um for a global totalitarian regime to be established and maybe then to be immune from overthrow. Um um so the paper doesn’t say that we should develop that kind of uh surveillance mesh. It just observes that if we are unlucky and the world is vulnerable in this respect then that might be the only thing that could prevent the world from being devastated and right now we don’t know whether the world is vulnerable in that way. You’ve warned that AI could let a society lock itself into permanent orthodoxy right. You also want global oversign of AI. How do you do you get one without the order? Well, I mean I think first of all this possibility of a permanent lock in might be possible without uh more advanced AI capabilities than than we already have. I think already now you could easily if you sort of are able to eaves drop on everybody’s conversations and read everybody’s messages. You have these mass surveillance, right? If you have AI kind of mining through that, I think you could probably build a pretty accurate view of what each person’s uh political opinions are or their views of of the government and so forth. Um, so with sufficient investment, you could imagine if if there were somebody already sitting as like if they were a world dictator and they had this technology and invested heavily, then maybe they would be able to leverage that to uh entrench themselves. Um um now I think um a longer term um um one would hope to arrange things in such a way that even if there are large scale political entities that there are some some sort of set of balance of power within those entities or entrenched rights or guarantees that would protect the members of those entities just as within nation states today. Even the the government is in some sense very powerful. They have like a whole army, right, with jet fighters and tanks and all of that, machine guns, way more than any individual citizens or any local militia. Nevertheless, in many cases, um they don’t devolve into dictatorships. Um and uh that’s one hope one could have for this future where uh things are increasingly run by AI. There might also be increasing um ways to arrange things in that world that are not really available today. Um cuz AI systems, if you imagine maybe some super intelligence being empowered to kind of run the show, it could be transparent and its motives could be engineered in a way that’s not the case with any human. So with a human you maybe appoint somebody to be you know um the leader of a country uh and you sort of based on what they’re saying on the campaign trail but that’s not really a guarantee to what they will subsequently do once they are in power. um w with an AI in principle once we solve alignment you might actually be able to create sort of a perfect ruler that didn’t have any self-interest and that could not lie and that was always going to remain um committed to upholding the constitution and so forth. So it’s kind of a wider design space of possible solutions once you have this kind of technology. I don’t really understand one thing because you’ve warned about AI risk for decades, but you also said it would be a catastrophe in super AI was never developed. Why must we build AI? Well, there are a lot of things that um have both risks if you do them and risks if you don’t do them. So take somebody who has um some advanced progressive disease maybe like congestive heart failure or something and they they maybe think that at some point they need an operation like a bypass surgery. Um, so it can both be true that going in for a bypass operation is risky and they might in fact die on the operating table and it also being true at the same time that um at some point they really need to have this bypass operation otherwise they’re going to die and these uh unfortunately the the the world is often such that there are complex but still this is just only one small part of the reality right I think exactly the same situation pertains with respect to um super intelligence and indeed other technologies that we develop that might you know you make progress in biotechnology then there could be quite dangerous capabilities being unlocked in terms of the capacity to design new kinds of viruses and pathogens um that will become easier as you get better instrumentation better understanding of how to design organisms um it’s also the case of course that we are hoping for great benefits from biomedical innovation. Um the world is very often complex like that in deeptopia right you describe a world where AI solves every problem is a solved life worth living. Um I think it uh could be extremely uh wonderful um on balance. Um so the book really dives into this. It’s it’s a deep pro question really once you really start to think through what it would mean um to attain technological maturity like if super intelligence truly succeeds. Um right now so much of our problems is oriented of our lives um are structured around trying to solve various problems. So you need to get a paycheck right to pay the bill. Um, then you need to take out the trash because otherwise your house starts to smell. And then you need to do that and you need to do that. You need to brush your teeth so that your teeth don’t rot. Um, and then you need to go to school so you can later get a job. Like there’s like this kind of set of instrumental necessities really give a kind of structure to our lives. Um, and a lot of that would go away at technological maturity. That would be easier. there would be shortcuts to all of these outcomes that wouldn’t um require uh your own investment of time and effort. Um and so then the question is what can remain in that situation and um I think once you sort of um think it all through I think there will be an existence that could be very very good. It will lack a few of the things that we currently think are important and valuable, but it would also enable a whole range of new things that could be even more. So, it would be a big um transformation and um a kind of uh would require a massive rethink of how we structure our lives. But ultimately I think there is something there at the other end of that process that that would be worth. If super intelligence manages our economy, our safety, our environment perfectly, we effectively become kept creatures is the ultimate destiny of humanity to become a well-loved golden retriever. Well, a golden retriever or just somebody who has a lot of money uh who doesn’t need to work themselves, but they have staff that helps them and then they spend their time doing the things they actually want to do instead. Um that might be quite attractive and I think a lot of people would jump at uh that if they had the chance. like your fifth um if if you had some rich uncle who died and bequeathed you $3 billion like would you just uh rip up the check or would you sort of cash it in? Most people would cash it. Um and here you have something even more extreme because there are a lot of things in today’s world that money can’t buy. Um for example um health only to a very limited degree. Can you improve your health with by spending money on it? Like the richest people also get old and sick and you know if they want to be fit they still have to go to the gym. That’s one example. Um many other things are currently just not purchasable at any price and that but they they would become available in in this scenario. Uh um um um yeah I I think in fact there there is a kind of tradition u say amongst the British aristocracy that having to work for a living is a kind of an unfortunate condition right like you’re not truly free if if you’re you don’t have control over your time um if you have to sell your hours to to make a living to get your daily Right? That’s a kind of a regrettable misfortune that many people had to live under. But the ideal condition is is to just have as much money as you need and then you spend your time engaging in politics or you’re talking with friends or you’re out in nature or you’re playing cards or you’re reading books or you’re doing what whatever whatever pleases you. Um and you’re the master of your own time. I think in some ways that’s like maybe a more dignified way of existing than than our current lot. Does Chad GPT or Claude have an inner life for you? That’s the question. At what point should we start worrying about the moral status of AI system? Probably a few years ago or something. Um I think we are currently um in a situation where at least we can’t be confident that some AI systems don’t already have moral status to various degrees and forms. I think it’s going to become increasingly plausible as the complexity and uh uh range of capabilities increase. Um so one way that this could be so I think is if they are sentient and have conscious experiences um and the ability to suffer I think that’s a sufficient condition for having moral status um and most people would agree with that. I think there might also be an alternative basis for having moral status which is maybe if you have a conception of self as existing through time um life goals that you’re hoping to achieve. Uh maybe the ability to form reciprocal relationships with with with other human beings. Um I think if you have some set of those kinds of capabilities that also plausibly makes it so that there are ways of treating you that would be wrong for your sake. not just because it would harm the AI’s owner or something. Um and um so I think this is one very important aspect of this challenge of navigating the transition to the machine intelligence era. Uh getting the ethics of digital minds right. Um, and it’s one thing to say affirm the principle that yes, they could be AIS, maybe they will have some forms of moral status and we should be nice to them. But then exactly what does that mean in practice? Like what precisely does this moral status amount to? What what do these different AIs want? What practically should we be doing differently if we want to be nice to AIS? that’s like a big open uh field um where there needs to be work from you know philosophers, AI researchers and and ultimately it will become like a political issue as well um where just there’s a lot of stuff that still needs to be thought through. You’ve been thinking about existential risk for 25 years. What did you get wrong? In many respects, um the changes are more in terms of um getting a a higher resolution view. It’s like when you’re loading um maybe like Google Maps over a slow connection like initially maybe you get like a very low res version of just kind of some pixelated um map of the landscape, right? And then as as more bits come down, it kind of gets all photorealistic. And similarly back in the ‘9s when I you know some of my early work uh like there’s like a kind of very um low res version. I could see AI uh likely being a key thing. Um I saw like probably serial progress that we had seen up to that point with Mor’s law was going to maybe last for another 10 15 years then slow down and that therefore super intelligence probably would take the form of some massively parallel computing system probably neural networks. I was obsessed with neural networks since even before that actually like from when I was like 15. You was or you are? I was uh and have been since that that seemed to me like on the path. Um so a lot of that but now of course we can see in much finer granularity like exactly how things are unfolding. Um, one thing that um wasn’t obvious to me um was that there was going to be this quite extended period of time where we have roughly human level AI systems that is AIS that can talk to us in natural language that you can have conversations with. um that have you know in many respects quite human psychologies and human foibballs um and that this period would last I mean it’s already been around for what like 3 four 5 years right and maybe will last a few more that’s quite an extended period of dealing with something that is kind of alien but still roughly humanish level um that that wasn’t clear I I there were scenarios where the takeoff would go slowly and and ones where it might happen more quickly. Somebody sitting tinkering with their you know computer figuring out the key algorithm and things kind of going f over a short period of time but and and there was some probability on on this this this kind of scenario as well but not an overwhelming probability relative to it either going much much slower or much faster more likely faster. In the past you’ve spar controversially with statements about intelligence. Do you believe that scientists should publish any statistical through? Probably there are areas where uh less progress is better than more progress in terms of our scientific understanding. Um, I mentioned we talked a little bit earlier about biological weapons research as being a a very plausible example of such an area where I might want to see less progress towards understanding how biological weapons would work and how they could be made. Um, and probably there are many other areas as well. It’s very difficult in general um, to know how to think about these. I had this paper uh introducing the concept of an information hazard. Um which is like so there are a lot of obvious ways in which false information and lies can be harmful, right? like you know somebody spews out false financial information, investors invest and they all lose their money and it’s like um but more interesting from a theoretical point of view are cases where true information uh can be harmful and the different ways in which that can happen. Um and it’s particularly salient in so like a lot of my earlier work at the future of humanity institute was kind of studying existential risks like different threats the biggest threats to the human species and to human civilization. Um we did sometimes run into u a very non-trivial question as to whether it would actually be good to publish on like because on the one hand we need to understand what the risks are in order to be able to better avoid them and take action, right? But you also don’t want to lay out in detail so much specificity about the risk that it basically creates a blueprint, right? For people to actualize it. Um so then you think well let’s just say something generic about the thing without spelling out all the details and you think but then well if you do the first step then some other eager academic who wants citations and who is clever enough given the early sort of general pointer to work out the details they’re going to then come in and write their paper filling out and then the whole thing fills out even if you sort of were a little bit um um careful in in the beginning. So then maybe you shouldn’t even point in the general directions where you think the risks lie. Um and it’s really hard. I mean we were wrestling a lot with this and uh um it because it’s really hard to know exactly that it’s like it involves such heavy judgment calls. I think what one can observe is that if there is a bias in science, it’s probably towards excessive publication and dissemination because that’s where the incentives are. um you get rewarded as as a scientist, an academic for publishing and getting citations. You don’t get rewarded for having been able to publish something but deciding not to because you think it would be too dangerous for the world. So the default expectation would be that given that scientists want prosperous careers, they probably are on the side of publishing too much. Nick, uh, if you could find answer for only one question about reality, what it would be? Only one? It’d be like a um a very long and compound question that uh you can sort of bake a lot of things into one question if you’re allowed sufficiently many words. You can choose two or three. Uh uh um I mean what what ultimately uh what I at least ultimately I guess is most interested in is is what I should be doing or not doing. So things that ultimately bring it down to sort of some immediately action relevant um um answer. Um, so if one can ask like would it be wise for me to try to like slow or to accelerate AI development? Like would it be wise to try to get governments more involved or less involved? Would it be like maybe some questions of that sort um would be have the highest information value? Um perhaps I I think if if I actually were given a genie that would answer uh some questions, I would want to maybe give it some more thought. Thank you very much for your time, Nick. Well, thank you for uh reaching out. It was honor.