How to Build AGI? (Ilya Sutskever)

00:00:00.000 | What do you think it takes to,

00:00:03.040 | let's talk about AGI a little bit.

00:00:04.720 | What do you think it takes to build a system

00:00:06.660 | of human level intelligence?

00:00:08.240 | We talked about reasoning,

00:00:09.840 | we talked about long-term memory,

00:00:11.580 | but in general, what does it take, do you think?

00:00:13.980 | Well, I can't be sure,

00:00:16.480 | but I think that deep learning plus maybe another

00:00:21.160 | small idea.

00:00:24.040 | Do you think self-play will be involved?

00:00:25.880 | So like you've spoken about the powerful mechanism

00:00:28.720 | of self-play where systems learn by

00:00:31.760 | sort of exploring the world in a competitive setting

00:00:36.960 | against other entities that are similarly skilled as them

00:00:40.860 | and so incrementally improve in this way.

00:00:43.340 | Do you think self-play will be a component

00:00:44.840 | of building an AGI system?

00:00:46.980 | Yeah, so what I would say to build AGI,

00:00:50.600 | I think is going to be deep learning plus some ideas,

00:00:55.320 | and I think self-play will be one of those ideas.

00:00:58.760 | I think that that is a very,

00:01:00.760 | self-play has this amazing property that it can surprise us

00:01:06.520 | in truly novel ways.

00:01:09.720 | For example,

00:01:10.840 | I mean, pretty much every self-play system,

00:01:16.680 | both our Dota bot,

00:01:18.560 | I don't know if OpenAI had a release about multi-agent

00:01:22.800 | where you had two little agents who were playing hide and seek

00:01:26.200 | and of course also AlphaZero.

00:01:28.560 | They were all produced surprising behaviors.

00:01:31.360 | They all produce behaviors that we didn't expect.

00:01:33.520 | They are creative solutions to problems.

00:01:36.200 | And that seems like an important part of AGI

00:01:39.040 | that our systems don't exhibit routinely right now.

00:01:42.520 | And so that's why I like this area,

00:01:45.240 | I like this direction because of its ability to surprise us.

00:01:47.880 | - To surprise us.

00:01:48.720 | And an AGI system would surprise us fundamentally.

00:01:51.520 | - Yes, and to be precise, not just a random surprise,

00:01:54.840 | but to find the surprising solution to a problem

00:01:58.240 | is also useful.

00:01:59.480 | - Right.

00:02:00.320 | Now, a lot of the self-play mechanisms have been used

00:02:03.840 | in the game context, or at least in the simulation context.

00:02:08.720 | How far along the path to AGI

00:02:15.400 | do you think will be done in simulation?

00:02:17.000 | How much faith, promise do you have in simulation

00:02:21.640 | versus having to have a system that operates

00:02:24.800 | in the real world, whether it's the real world

00:02:27.760 | of digital real-world data or real-world,

00:02:30.960 | like actual physical world of robotics?

00:02:33.520 | - I don't think it's an either/or.

00:02:35.320 | I think simulation is a tool and it helps.

00:02:37.840 | It has certain strengths and certain weaknesses

00:02:39.960 | and we should use it.

00:02:41.760 | - Yeah, but, okay, I understand that.

00:02:44.720 | That's true, but one of the criticisms of self-play,

00:02:53.000 | one of the criticisms of reinforcement learning

00:02:55.040 | is one of the,

00:02:55.880 | its current power, its current results,

00:03:01.320 | while amazing, have been demonstrated

00:03:03.160 | in its simulated environments,

00:03:05.060 | or very constrained physical environments.

00:03:06.660 | Do you think it's possible to escape them,

00:03:09.440 | escape the simulated environments

00:03:11.040 | and be able to learn in non-simulated environments?

00:03:13.680 | Or do you think it's possible to also just simulate

00:03:17.300 | in a photorealistic and physics realistic way

00:03:21.400 | the real world in a way that we can solve real problems

00:03:24.040 | with self-play in simulation?

00:03:27.000 | - So I think that transfer from simulation

00:03:29.960 | to the real world is definitely possible

00:03:31.960 | and has been exhibited many times by many different groups.

00:03:36.320 | It's been especially successful in vision.

00:03:38.920 | Also, open AI in the summer has demonstrated a robot hand

00:03:42.920 | which was trained entirely in simulation

00:03:45.520 | in a certain way that allowed

00:03:46.920 | for seem-to-real transfer to occur.

00:03:50.120 | - Is this for the Rubik's Cube?

00:03:51.680 | - Yes, right.

00:03:52.960 | - I wasn't aware that was trained in simulation.

00:03:54.960 | - It was trained in simulation entirely.

00:03:57.280 | - Really, so it wasn't in the physical,

00:03:59.680 | the hand wasn't trained?

00:04:01.240 | - No.

00:04:02.080 | 100% of the training was done in simulation.

00:04:05.080 | And the policy that was learned in simulation

00:04:07.160 | was trained to be very adaptive.

00:04:09.240 | So adaptive that when you transfer it,

00:04:11.200 | it could very quickly adapt to the physical world.

00:04:14.200 | - So the kind of perturbations with the giraffe

00:04:17.640 | or whatever the heck it was,

00:04:19.140 | were those part of the simulation?

00:04:22.120 | - Well, the simulation was generally,

00:04:24.400 | so the simulation was trained to be robust

00:04:27.320 | to many different things,

00:04:28.400 | but not the kind of perturbations we've had in the video.

00:04:30.880 | So it's never been trained with a glove,

00:04:32.920 | it's never been trained with a stuffed giraffe.

00:04:37.280 | - So in theory, these are novel perturbations.

00:04:39.560 | - Correct, it's not in theory, in practice.

00:04:41.960 | - That those are novel perturbations?

00:04:44.040 | Well, that's okay.

00:04:46.240 | That's a clean, small scale, but clean example

00:04:49.700 | of a transfer from the simulated world

00:04:51.100 | to the physical world.

00:04:52.420 | - Yeah, and I will also say that I expect

00:04:54.580 | the transfer capabilities of deep learning

00:04:56.540 | to increase in general.

00:04:58.460 | And the better the transfer capabilities are,

00:05:00.820 | the more useful simulation will become.

00:05:02.780 | Because then you could take,

00:05:05.540 | you could experience something in simulation

00:05:08.820 | and then learn a moral of the story,

00:05:10.620 | which you could then carry with you to the real world.

00:05:13.820 | As humans do all the time when they play computer games.

00:05:17.240 | - So let me ask sort of an embodied question,

00:05:22.040 | staying on AGI for a sec.

00:05:23.880 | Do you think AGI says that we need to have a body?

00:05:28.000 | We need to have some of those human elements

00:05:29.840 | of self-awareness, consciousness,

00:05:33.280 | sort of fear of mortality, sort of self-preservation

00:05:36.960 | in the physical space, which comes with having a body?

00:05:40.640 | - I think having a body will be useful.

00:05:42.720 | I don't think it's necessary,

00:05:44.620 | but I think it's very useful to have a body for sure,

00:05:46.540 | because you can learn a whole new,

00:05:49.180 | you can learn things which cannot be learned without a body.

00:05:52.800 | But at the same time, I think that you can,

00:05:54.800 | if you don't have a body, you could compensate for it

00:05:57.220 | and still succeed.

00:05:58.900 | - You think so?

00:05:59.740 | - Yes.

00:06:00.560 | Well, there is evidence for this.

00:06:01.400 | For example, there are many people who were born deaf

00:06:03.620 | and blind and they were able to compensate

00:06:06.860 | for the lack of modalities.

00:06:08.540 | I'm thinking about Helen Keller specifically.

00:06:11.860 | - So even if you're not able to physically interact

00:06:14.100 | with the world, and if you're not able to,

00:06:17.220 | I mean, I actually was getting at,

00:06:19.040 | maybe let me ask on the more particular,

00:06:22.960 | I'm not sure if it's connected to having a body or not,

00:06:25.620 | but the idea of consciousness,

00:06:28.140 | and a more constrained version of that is self-awareness.

00:06:31.540 | Do you think an AGI system should have consciousness?

00:06:34.840 | We can't define, whatever the heck you think consciousness is.

00:06:39.700 | - Yeah, hard question to answer,

00:06:41.900 | given how hard it is to define it.

00:06:43.600 | - Do you think it's useful to think about?

00:06:46.780 | - I mean, it's definitely interesting.

00:06:48.700 | It's fascinating.

00:06:50.180 | I think it's definitely possible

00:06:52.140 | that our systems will be conscious.

00:06:54.220 | - Do you think that's an emergent thing

00:06:55.380 | that just comes from,

00:06:56.740 | do you think consciousness could emerge

00:06:58.100 | from the representation that's stored within your networks?

00:07:01.160 | So like that it naturally just emerges

00:07:03.300 | when you become more and more,

00:07:05.420 | you're able to represent more and more of the world.

00:07:07.340 | - Well, I'd say I'd make the following argument,

00:07:09.100 | which is,

00:07:09.940 | humans are conscious,

00:07:14.100 | and if you believe that artificial neural nets

00:07:16.380 | are sufficiently similar to the brain,

00:07:19.860 | then there should at least exist artificial neural nets

00:07:23.020 | we should be conscious to.

00:07:24.560 | - You're leaning on that existence proof pretty heavily.

00:07:26.900 | Okay.

00:07:27.740 | - But that's the best answer I can give.

00:07:32.380 | - No, I know, I know, I know.

00:07:36.300 | There's still an open question

00:07:37.420 | if there's not some magic in the brain that we're not,

00:07:41.100 | I mean, I don't mean a non-materialistic magic,

00:07:43.940 | but that the brain might be a lot more complicated

00:07:48.100 | and interesting than we give it credit for.

00:07:50.180 | - If that's the case, then it should show up.

00:07:52.820 | And at some point,

00:07:54.020 | - At some point.

00:07:54.860 | - We will find out that we can't continue to make progress.

00:07:56.900 | But I think it's unlikely.

00:07:59.060 | - So we talk about consciousness,

00:08:00.460 | but let me talk about another poorly defined concept

00:08:02.700 | of intelligence.

00:08:03.760 | Again, we've talked about reasoning,

00:08:07.140 | we've talked about memory.

00:08:08.380 | What do you think is a good test of intelligence for you?

00:08:11.940 | Are you impressed by the test that Alan Turing formulated

00:08:15.980 | with the imitation game of natural language?

00:08:18.860 | Is there something in your mind

00:08:21.400 | that you will be deeply impressed by

00:08:24.540 | if a system was able to do?

00:08:26.720 | - I mean, lots of things.

00:08:28.300 | There's a certain frontier of capabilities today.

00:08:33.540 | And there exists things outside of that frontier.

00:08:37.220 | And I would be impressed by any such thing.

00:08:39.260 | For example, I would be impressed by a deep learning system

00:08:44.260 | which solves a very pedestrian task,

00:08:47.540 | like machine translation or computer vision task

00:08:50.000 | or something which never makes mistake

00:08:53.700 | a human wouldn't make under any circumstances.

00:08:57.580 | I think that is something

00:08:58.860 | which have not yet been demonstrated

00:09:00.340 | and I would find it very impressive.

00:09:03.040 | - Yeah, so right now they make mistakes in different,

00:09:05.180 | they might be more accurate than human beings,

00:09:06.900 | but they still, they make a different set of mistakes.

00:09:09.420 | - So my, I would guess that a lot of the skepticism

00:09:13.740 | that some people have about deep learning

00:09:16.080 | is when they look at their mistakes and they say,

00:09:17.580 | "Well, those mistakes, they make no sense."

00:09:20.540 | Like if you understood the concept,

00:09:21.920 | you wouldn't make that mistake.

00:09:23.420 | And I think that changing that would be,

00:09:27.200 | that would inspire me, that would be yes,

00:09:30.300 | this is progress.

00:09:32.820 | - Yeah, that's a really nice way to put it.

00:09:35.700 | But I also just don't like that human instinct

00:09:38.780 | to criticize a model as not intelligent.

00:09:41.780 | That's the same instinct as we do

00:09:43.420 | when we criticize any group of creatures as the other.

00:09:48.020 | Because it's very possible that GPT-2

00:09:53.740 | is much smarter than human beings at many things.

00:09:56.660 | - That's definitely true.

00:09:57.840 | It is a lot more breadth of knowledge.

00:09:59.620 | - Yes, breadth of knowledge.

00:10:01.260 | And even perhaps depth on certain topics.

00:10:05.220 | - It's kind of hard to judge what depth means,

00:10:08.620 | but there's definitely a sense in which

00:10:11.420 | humans don't make mistakes, these models do.

00:10:14.780 | - Yes, the same is applied to autonomous vehicles.

00:10:18.060 | The same is probably gonna continue being applied

00:10:19.960 | to a lot of artificial intelligence systems.

00:10:22.020 | We find, this is the annoying thing,

00:10:24.380 | this is the process of, in the 21st century,

00:10:27.060 | the process of analyzing the progress of AI

00:10:29.740 | is the search for one case where the system fails

00:10:33.620 | in a big way where humans would not.

00:10:37.260 | And then many people writing articles about it.

00:10:40.940 | And then broadly, the public generally gets convinced

00:10:45.060 | that the system is not intelligent.

00:10:46.860 | And we pacify ourselves by thinking it's not intelligent

00:10:50.140 | because of this one anecdotal case.

00:10:52.260 | And this seems to continue happening.

00:10:54.820 | - Yeah, I mean, there is truth to that.

00:10:57.180 | Although I'm sure that plenty of people

00:10:58.420 | are also extremely impressed by the systems

00:11:00.100 | that exist today.

00:11:01.100 | But I think this connects to the earlier point

00:11:02.780 | we discussed that it's just confusing

00:11:05.260 | to judge progress in AI.

00:11:07.340 | - Yeah.

00:11:08.180 | - And you have a new robot demonstrating something.

00:11:11.020 | How impressed should you be?

00:11:13.020 | And I think that people will start to be impressed

00:11:16.260 | once AI starts to really move the needle on the GDP.

00:11:19.640 | - So you're one of the people that might be able

00:11:22.300 | to create an AGI system here, not you, but you and OpenAI.

00:11:27.100 | If you do create an AGI system

00:11:29.300 | and you get to spend sort of the evening with it, him, her,

00:11:34.300 | what would you talk about, do you think?

00:11:37.060 | - The very first time?

00:11:39.420 | - First time.

00:11:40.260 | - Well, the first time I would just ask

00:11:43.020 | all kinds of questions and try to make it,

00:11:44.820 | to get it to make a mistake.

00:11:45.980 | And I would be amazed that it doesn't make mistakes

00:11:48.380 | and just keep asking broad questions.

00:11:53.380 | - What kind of questions do you think,

00:11:55.200 | would they be factual or would they be personal,

00:11:59.380 | emotional, psychological, what do you think?

00:12:01.900 | - All of the above.

00:12:03.700 | - Would you ask for advice?

00:12:07.540 | - Definitely.

00:12:08.380 | I mean, why would I limit myself

00:12:11.860 | talking to a system like this?

00:12:13.420 | - Now, again, let me emphasize the fact

00:12:16.380 | that you truly are one of the people

00:12:18.060 | that might be in the room where this happens.

00:12:21.500 | So let me ask sort of a profound question about,

00:12:26.500 | I just talked to a Stalin historian.

00:12:28.640 | (laughs)

00:12:30.280 | Been talking to a lot of people who are studying power.

00:12:33.440 | Abraham Lincoln said, "Nearly all men can stand adversity,

00:12:37.960 | "but if you want to test a man's character, give him power."

00:12:41.640 | I would say the power of the 21st century, maybe the 22nd,

00:12:46.640 | but hopefully the 21st, would be the creation

00:12:49.520 | of an AGI system and the people who have control,

00:12:53.700 | direct possession and control of the AGI system.

00:12:56.560 | So what do you think, after spending that evening

00:13:00.580 | having a discussion with the AGI system,

00:13:04.460 | what do you think you would do?

00:13:06.020 | - Well, the ideal world I'd like to imagine

00:13:09.420 | is one where humanity, alike,

00:13:15.900 | the board members of a company,

00:13:19.460 | where the AGI is the CEO.

00:13:21.020 | So it would be, I would like,

00:13:27.260 | the picture which I would imagine

00:13:28.940 | is you have some kind of different entities,

00:13:32.700 | different countries or cities,

00:13:34.900 | and the people that leave there vote

00:13:36.380 | for what the AGI that represents them should do,

00:13:39.420 | and an AGI that represents them goes and does it.

00:13:41.700 | I think a picture like that, I find very appealing.

00:13:46.700 | And you could have multiple,

00:13:47.540 | you would have an AGI for a city, for a country,

00:13:49.660 | and it would be trying to, in effect,

00:13:54.300 | take the democratic process to the next level.

00:13:56.380 | - And the board can always fire the CEO.

00:13:58.980 | - Essentially, press the reset button, say.

00:14:01.020 | - Press the reset button.

00:14:01.860 | - Rerandomize the parameters.

00:14:03.260 | - Well, let me sort of, that's actually,

00:14:06.300 | okay, that's a beautiful vision, I think,

00:14:09.380 | as long as it's possible to press the reset button.

00:14:12.720 | Do you think it will always be possible

00:14:15.300 | to press the reset button?

00:14:16.680 | - So I think that it's definitely will be possible to build.

00:14:20.720 | So you're talking,

00:14:23.260 | so the question that I really understand from you is,

00:14:26.940 | will humans or,

00:14:29.580 | humans people have control over the AI systems that they build?

00:14:34.540 | - Yes.

00:14:35.380 | - And my answer is, it's definitely possible

00:14:37.620 | to build AI systems which will want

00:14:39.860 | to be controlled by their humans.

00:14:42.140 | - Wow, that's part of their,

00:14:44.340 | so it's not that just they can't help it be controlled,

00:14:46.500 | but that's,

00:14:47.340 | they exist,

00:14:52.040 | one of the objectives of their existence is to be controlled

00:14:54.800 | in the same way that human parents

00:14:58.040 | generally want to help their children,

00:15:02.740 | they want their children to succeed.

00:15:04.720 | It's not a burden for them.

00:15:06.320 | They are excited to help the children and to feed them

00:15:09.640 | and to dress them and to take care of them.

00:15:12.960 | And I believe with high conviction

00:15:16.620 | that the same will be possible for an AGI.

00:15:19.220 | It will be possible to program an AGI,

00:15:20.820 | to design it in such a way

00:15:22.020 | that it will have a similar deep drive,

00:15:25.140 | that it will be delighted to fulfill

00:15:27.380 | and the drive will be to help humans flourish.

00:15:30.240 | - But let me take a step back to that moment

00:15:34.260 | where you create the AGI system.

00:15:35.780 | I think this is a really crucial moment.

00:15:39.300 | And between that moment and the Democratic board members

00:15:44.300 | with the AGI at the head,

00:15:48.000 | there has to be a relinquishing of power.

00:15:52.120 | So as George Washington,

00:15:54.200 | despite all the bad things he did,

00:15:56.760 | one of the big things he did is he relinquished power.

00:15:59.640 | He, first of all, didn't want to be president.

00:16:02.440 | And even when he became president,

00:16:04.040 | he gave, he didn't keep just serving

00:16:06.240 | as most dictators do for indefinitely.

00:16:09.340 | Do you see yourself being able to relinquish control

00:16:14.340 | over an AGI system,

00:16:16.540 | given how much power you can have over the world?

00:16:19.500 | At first financial, just make a lot of money, right?

00:16:22.980 | And then control by having possession of this AGI system.

00:16:27.260 | - I'd find it trivial to do that.

00:16:29.260 | I'd find it trivial to relinquish this kind of power.

00:16:31.700 | I mean, you know, the kind of scenario you are describing

00:16:35.300 | sounds terrifying to me.

00:16:37.620 | That's all.

00:16:39.220 | I would absolutely not want to be in that position.

00:16:42.640 | - Do you think you represent the majority

00:16:45.900 | or the minority of people in the AI community?

00:16:49.660 | - Well, I mean.

00:16:50.980 | - It's an open question and an important one.

00:16:54.020 | Are most people good is another way to ask it.

00:16:56.740 | - So I don't know if most people are good,

00:16:59.580 | but I think that when it really counts,

00:17:04.580 | people can be better than we think.

00:17:07.320 | - That's beautifully put, yeah.

00:17:09.540 | Are there specific mechanism you can think of

00:17:11.740 | of aligning AI gene values to human values?

00:17:14.840 | Is that, do you think about these problems

00:17:16.940 | of continued alignment as we develop the AI systems?

00:17:20.580 | - Yeah, definitely.

00:17:21.640 | In some sense, the kind of question which you are asking is,

00:17:27.580 | so if I were to translate the question to today's terms,

00:17:30.940 | it would be a question about how to get an RL agent

00:17:36.880 | that's optimizing a value function which itself is learned.

00:17:41.440 | And if you look at humans, humans are like that

00:17:43.440 | because the reward function, the value function of humans

00:17:46.540 | is not external, it is internal.

00:17:49.080 | - That's right.

00:17:50.400 | - And there are definite ideas

00:17:54.140 | of how to train a value function.

00:17:57.040 | Basically an objective, you know,

00:17:59.360 | and as objective as possible perception system

00:18:01.800 | that will be trained separately.

00:18:05.700 | To recognize, to internalize human judgments

00:18:10.140 | on different situations.

00:18:12.260 | And then that component would then be integrated

00:18:14.900 | as the base value function

00:18:16.780 | for some more capable RL system.

00:18:19.300 | You could imagine a process like this.

00:18:20.860 | I'm not saying this is the process,

00:18:22.700 | I'm saying this is an example

00:18:24.060 | of the kind of thing you could do.

00:18:25.980 | (silence)

00:18:28.140 | (silence)

00:18:30.300 | (silence)

00:18:32.460 | (silence)

00:18:34.620 | (silence)

00:18:36.780 | (silence)

00:18:38.940 | (silence)

00:18:41.100 | [BLANK_AUDIO]

How to Build AGI? (Ilya Sutskever) | AI Podcast Clips

Chapters