- Let's talk about superintelligence, at least for a little bit. And let's start at the basics. What to you is intelligence? - Yeah, not to get too stuck with the definitional question. I mean, the common sense to understand, like the ability to solve complex problems, to learn from experience, to plan, to reason, some combination of things like that.

- Is consciousness mixed up into that or no? Is consciousness mixed up into that or is it- - Well, I think it could be fairly intelligent, at least without being conscious probably. - So then what is superintelligence? - Yeah, that would be like something that was much more- - Of that.

- Had much more general cognitive capacity than we humans have. So if we talk about general superintelligence, it would be much faster learner be able to reason much better, make plans that are more effective at achieving its goals, say in a wide range of complex, challenging environments. - In terms of, as we turn our eye to the idea of sort of existential threats from superintelligence, do you think superintelligence has to exist in the physical world or can it be digital only?

Sort of, we think of our general intelligence as us humans, as an intelligence that's associated with the body that's able to interact with the world, that's able to affect the world directly with physically. - I mean, digital only is perfectly fine, I think. I mean, it's physical in the sense that obviously the computers and the memories are physical.

- But it's capability to affect the world sort of- - Could be very strong even if it has a limited set of actuators, if it can type text on the screen or something like that, that would be, I think, ample. - So in terms of the concerns of existential threat of AI, how can an AI system that's in the digital world have existential risk sort of, and what are the attack vectors for a digital system?

- Well, I mean, I guess maybe to take one step back, so I should emphasize that I also think there's this huge positive potential from machine intelligence, including superintelligence. And I wanna stress that because some of my writing has focused on what can go wrong. And when I wrote the book "Superintelligence," at that point, I felt that there was a kind of neglect of what would happen if AI succeeds, and in particular, a need to get a more granular understanding of where the pitfalls are so we can avoid them.

I think that since the book came out in 2014, there has been a much wider recognition of that, and a number of research groups are now actually working on developing, say, AI alignment techniques and so on and so forth. So I'd like, yeah, I think now it's important to make sure we bring back onto the table the upside as well.

- And there's a little bit of a neglect now on the upside, which is, I mean, if you look at, I was talking to a friend, if you look at the amount of information that is available, or people talking, or people being excited about the positive possibilities of general intelligence, that's not, it's far outnumbered by the negative possibilities in terms of our public discourse.

- Possibly, yeah. It's hard to measure it, but-- - What are, can you linger on that for a little bit? What are some, to you, possible big positive impacts of general intelligence, superintelligence? - Well, I mean, superintelligence, because I tend to also wanna distinguish these two different contexts of thinking about AI and AI impacts, the kind of near-term and long-term, if you want, both of which I think are legitimate things to think about, and people should discuss both of them, but they are different, and they often get mixed up, and then I get, you get confusion.

Like, I think you get simultaneously, like maybe an overhyping of the near-term and an underhyping of the long-term, and so I think as long as we keep them apart, we can have, like, two good conversations, but, or we can mix them together and have one bad conversation. - Can you clarify just the two things we're talking about, the near-term and the long-term?

What are the distinctions? - Well, it's a blurry distinction, but say the things I wrote about in this book, superintelligence, long-term, things people are worrying about today with, I don't know, algorithmic discrimination, or even things, self-driving cars and drones and stuff, more near-term. And then, of course, you could imagine some medium-term where they kind of overlap and one evolves into the other.

But at any rate, I think both, yeah, the issues look kind of somewhat different depending on which of these contexts. - So I think it would be nice if we can talk about the long-term, and think about a positive impact or a better world because of the existence of the long-term superintelligence.

Do you have views of such a world? - Yeah, I mean, I guess it's a little hard to articulate because it seems obvious that the world has a lot of problems as it currently stands. And it's hard to think of any one of those which it wouldn't be useful to have, like, a friendly aligned superintelligence working on.

- So from health to the economic system to be able to sort of improve the investment and trade and foreign policy decisions, all that kind of stuff. - All that kind of stuff and a lot more. - I mean, what's the killer app? - Well, I don't think there is one.

I think AI, especially artificial general intelligence, is really the ultimate general purpose technology. So it's not that there is this one problem, this one area where it will have a big impact, but if and when it succeeds, it will really apply across the board in all fields where human creativity and intelligence and problem solving is useful, which is pretty much all fields, right?

The thing that it would do is give us a lot more control over nature. It wouldn't automatically solve the problems that arise from conflict between humans, fundamentally political problems. Some subset of those might go away if we just had more resources and cooler tech, but some subset would require coordination that is not automatically achieved just by having more technological capability.

But anything that's not of that sort, I think you just get like an enormous boost with this kind of cognitive technology once it goes all the way. Now, again, that doesn't mean I'm like thinking, oh, people don't recognize what's possible with current technology and like sometimes things get overhyped, but I mean, those are perfectly consistent views to hold, the ultimate potential being enormous.

And then it's a very different question of how far are we from that or what can we do with near-term technology? - Yeah, so what's your intuition about the idea of intelligence explosion? So there's this, you know, when you start to think about that leap from the near-term to the long-term, the natural inclination, like for me, sort of building machine learning systems today it seems like it's a lot of work to get to general intelligence, but there's some intuition of exponential growth, of exponential improvement, of intelligence explosion.

Can you maybe try to elucidate, try to talk about what's your intuition about the possibility of a intelligence explosion, that it won't be this gradual, slow process, there might be a phase shift? - Yeah, I think it's, we don't know how explosive it will be. I think for what it's worth, seems fairly likely to me that at some point there will be some intelligence explosion, like some period of time where progress in AI becomes extremely rapid.

Roughly in the area where you might say it's kind of human-ish equivalent in core cognitive faculties, that the concept of human equivalent, like it starts to break down when you look too closely at it and just how explosive does something have to be for it to be called an intelligence explosion?

Like, does it have to be like overnight literally, or a few years? But overall, I guess, if you plotted the opinions of different people in the world, I guess that would be somewhat more probability towards the intelligence explosion scenario than probably the average AI researcher, I guess. - So, and then the other part of the intelligence explosion, or just, forget explosion, just progress, is once you achieve that gray area of human-level intelligence, is it obvious to you that we should be able to proceed beyond it to get to super intelligence?

- Yeah, that seems, I mean, as much as any of these things can be obvious, given we've never had one, people have different views, smart people have different views, it's like there's some degree of uncertainty that always remains for any big, futuristic, philosophical, grand question that just we realize humans are fallible, especially about these things.

But it does seem, as far as I'm judging things, based on my own impressions, that it seems very unlikely that that would be a ceiling at or near human cognitive capacity. - But, and that's such a, I don't know, that's such a special moment. It's both terrifying and exciting to create a system that's beyond our intelligence.

So, maybe you can step back and say, like, how does that possibility make you feel? That we can create something, it feels like there's a line beyond which it steps, it'll be able to outsmart you, and therefore it feels like a step where we lose control. - Well, I don't think the latter follows, that is, you could imagine, and in fact, this is what a number of people are working towards, making sure that we could ultimately project higher levels of problem-solving ability while still making sure that they are aligned, like they are in the service of human values.

I mean, so, losing control, I think, is not a given that that would happen. Now, you asked how it makes you feel. I mean, to some extent, I've lived with this for so long, since as long as I can remember, being an adult or even a teenager, it seemed to me obvious that at some point, AI will succeed.

- And so, I actually misspoke, I didn't mean control. I meant, because the control problem is an interesting thing and I think the hope is, at least we should be able to maintain control over systems that are smarter than us, but we do lose our specialness. It's sort of, we'll lose our place as the smartest, coolest thing on Earth.

And there's an ego involved with that, that humans aren't very good at dealing with. I mean, I value my intelligence as a human being. It seems like a big transformative step to realize there's something out there that's more intelligent. I mean, you don't see that as such a fundamental-- - Yeah, I think, yes, a lot, I think it would be small.

I mean, I think there are already a lot of things out there that are, I mean, certainly if you think the universe is big, there's gonna be other civilizations that already have super intelligences or that just naturally have brains the size of beach balls and are like completely leaving us in the dust.

And we haven't come face to face with that. - We haven't come face to face, but I mean, that's an open question, what would happen in a kind of post-human world, like how much day to day would these super intelligences be involved in the lives of ordinary? I mean, you could imagine some scenario where it would be more like a background thing that would help protect against some things, but you wouldn't, like there wouldn't be this intrusive kind of like making you feel bad by like making clever jokes on your expense.

Like there's like all sorts of things that maybe in the human context would feel awkward about that. You don't wanna be the dumbest kid in your class, everybody picks it. Like a lot of those things, maybe you need to abstract away from, if you're thinking about this context where we have infrastructure that is in some sense beyond any or all humans.

I mean, it's a little bit like say the scientific community as a whole, if you think of that as a mind, it's a little bit of metaphor, but I mean, obviously it's gotta be like way more capacious than any individual. So in some sense, there is this mind like thing already out there that's just vastly more intelligent than a new individual is.

And we think, okay, that's, you just accept that as a fact. - That's the basic fabric of our existence is there's a super intelligent. - Yeah, you get used to a lot of. - I mean, there's already Google and Twitter and Facebook, these recommender systems that are the basic fabric of our, I could see them becoming, I mean, do you think of the collective intelligence of these systems as already perhaps reaching super intelligence level?

- Well, I mean, so here it comes to this, the concept of intelligence and the scale and what human level means. The kind of vagueness and indeterminacy of those concepts starts to dominate how you would answer that question. So like say the Google search engine has a very high capacity of a certain kind, like remembering and retrieving information, particularly like text or images that you have a kind of string, a word string key, obviously superhuman at that, but a vast set of other things it can't even do at all, not just not do well.

So you have these current AI systems that are superhuman in some limited domain and then like radically subhuman in all other domains. Same with a chess, like are just a simple computer that can multiply really large numbers, right? So it's gonna have this like one spike of super intelligence and then a kind of a zero level of capability across all other cognitive fields.

- Yeah, I don't necessarily think the generalness, I mean, I'm not so attached to it, but I could sort of, it's a gray area and it's a feeling, but to me sort of alpha zero is somehow much more intelligent, much, much more intelligent than Deep Blue. And to say which domain, well, you could say, well, these are both just board game, they're both just able to play board games, who cares if they're gonna do better or not?

But there's something about the learning, the self play-- - The learning, yeah. - That makes it, crosses over into that land of intelligence that doesn't necessarily need to be general. In the same way, Google is much closer to Deep Blue currently in terms of its search engine-- - Yeah.

- Than it is to sort of the alpha zero. And the moment it becomes, and the moment these recommender systems really become more like alpha zero, but being able to learn a lot without the constraints of being heavily constrained by human interaction, that seems like a special moment in time.

- I mean, certainly learning ability seems to be an important facet of general intelligence. - Right. - That you can take some new domain that you haven't seen before, and you weren't specifically pre-programmed for, and then figure out what's going on there, and eventually become really good at it.

So that's something alpha zero has much more of than Deep Blue had. And in fact, I mean, systems like alpha zero can learn, not just Go, but other, in fact, probably beat Deep Blue in chess and so forth. Right? - Yeah, not just Deep Blue. - So you do see this-- - Destroy Deep Blue.

- This general, and so it matches the intuition. We feel it's more intelligent, and it also has more of this general purpose learning ability. And if we get systems that have even more general purpose learning ability, it might also trigger an even stronger intuition that they are actually starting to get smart.

- So if you were to pick a future, what do you think a utopia looks like with AGI systems? Is it the neural link, brain-computer interface world, where we're kind of really closely interlinked with AI systems? Is it possibly where AGI systems replace us completely while maintaining the values and the consciousness?

Is it something like it's a completely invisible fabric, like you mentioned, a society where it's just AIDS and a lot of stuff that we do, like curing diseases and so on? What is utopia if you get to pick? - Yeah, I mean, it's a good question, and a deep and difficult one.

I'm quite interested in it. I don't have all the answers yet, but, or might never have, but I think there are some different observations one can make. One is if this scenario actually did come to pass, it would open up this vast space of possible modes of being. On one hand, material and resource constraints would just be expanded dramatically.

So there would be a lot of, a big pie, let's say, right? Also, it would enable us to do things, including to ourselves, or like that it would just open up this much larger design space and options space than we have ever had access to in human history. So I think two things follow from that.

One is that we probably would need to make a fairly fundamental rethink of what ultimately we value, like think things through more from first principles. The context would be so different from the familiar that we could have just take what we've always been doing and then like, oh, well, we have this cleaning robot that cleans the dishes in the sink.

And a few other small things. And like, I think we would have to go back to first principles. - So even from the individual level, go back to the first principles of what is the meaning of life, what is happiness, what is fulfillment? - Yeah. And then also connected to this large space of resources is that it would be possible.

And I think something we should aim for is to do well by the lights of more than one value system. That is, we wouldn't have to choose only one value criterion and say, we're gonna do something that scores really high on the metric of say hedonism. And then it's like a zero by other criteria, like kind of wire headed brain Cinebat.

And it's like a lot of pleasure, that's good. But then like no beauty, no achievement. Or pick it up. I think to some significant, not unlimited sense, but a significant sense, it would be possible to do very well by many criteria. Like maybe you could get like 98% of the best according to several criteria at the same time, given this great expansion of the option space.

And so- - So have competing value systems, competing criteria as a sort of forever, just like our Democrat versus Republican, there seems to be this always multiple parties that are useful for our progress in society, even though it might seem dysfunctional inside the moment, but having the multiple value systems seems to be beneficial for, I guess, a balance of power.

- So that's, yeah, not exactly what I have in mind that it's, well, although it can be, maybe in an indirect way it is. But that if you had the chance to do something that scored well on several different metrics, our first instinct should be to do that rather than immediately leap to the thing, which ones of these value systems are we gonna screw over?

Like I think our first instinct, let's first try to do very well by all of them. Then it might be that you can't get 100% of all, and you would have to then like have the hard conversation about which one will only get 97%. - There you go, there's my cynicism that all of existence is always a trade-off.

But you say, maybe it's not such a bad trade-off. Let's first at least try- - Well, this would be a distinctive context in which at least some of the constraints would be removed. - I'll leave you there. - So there's probably still be trade-offs in the end. It's just that we should first make sure we at least take advantage of this abundance.

So in terms of thinking about this, like, yeah, one should think, I think in this kind of frame of mind of generosity and inclusiveness to different value systems and see how far one can get there first. And I think one could do something that would be very good according to many different criteria.

(laughs) (upbeat music) (upbeat music) (upbeat music) (upbeat music) (upbeat music) (upbeat music)