Marcus Hutter: Universal Artificial Intelligence, AIXI, and AGI

00:00:00.000 | The following is a conversation with Marcus Hutter,

00:00:03.480 | senior research scientist at Google DeepMind.

00:00:06.680 | Throughout his career of research,

00:00:08.280 | including with Juergen Schmidhuber and Shane Legg,

00:00:11.600 | he has proposed a lot of interesting ideas

00:00:13.920 | in and around the field of artificial general intelligence,

00:00:17.480 | including the development of AIXI, spelled A-I-X-I, model,

00:00:22.480 | which is a mathematical approach to AGI

00:00:25.360 | that incorporates ideas of Kolmogorov complexity,

00:00:28.840 | Solomonov induction, and reinforcement learning.

00:00:32.000 | In 2006, Marcus launched the 50,000 Euro Hutter Prize

00:00:38.040 | for Lossless Compression of Human Knowledge.

00:00:41.160 | The idea behind this prize is that the ability

00:00:43.700 | to compress well is closely related to intelligence.

00:00:47.880 | This, to me, is a profound idea.

00:00:50.320 | Specifically, if you can compress the first 100 megabytes

00:00:54.880 | or one gigabyte of Wikipedia better than your predecessors,

00:00:58.320 | your compressor likely has to also be smarter.

00:01:02.160 | The intention of this prize is to encourage

00:01:04.240 | the development of intelligent compressors

00:01:06.400 | as a path to AGI.

00:01:08.360 | In conjunction with his podcast release just a few days ago,

00:01:13.240 | Marcus announced a 10X increase

00:01:15.520 | in several aspects of this prize,

00:01:17.440 | including the money, to 500,000 Euros.

00:01:22.440 | The better your compressor works

00:01:24.160 | relative to the previous winners,

00:01:26.040 | the higher fraction of that prize money

00:01:27.680 | is awarded to you.

00:01:29.400 | You can learn more about it

00:01:30.800 | if you Google simply Hutter Prize.

00:01:34.060 | I'm a big fan of benchmarks for developing AI systems,

00:01:38.080 | and the Hutter Prize may indeed be one

00:01:40.600 | that will spark some good ideas for approaches

00:01:43.120 | that will make progress on the path of developing AGI systems.

00:01:47.840 | This is the Artificial Intelligence Podcast.

00:01:50.520 | If you enjoy it, subscribe on YouTube,

00:01:52.720 | give it five stars on Apple Podcasts,

00:01:54.680 | support it on Patreon,

00:01:56.000 | or simply connect with me on Twitter @LexFriedman,

00:01:59.240 | spelled F-R-I-D-M-A-N.

00:02:02.600 | As usual, I'll do one or two minutes of ads now,

00:02:05.480 | and never any ads in the middle

00:02:06.960 | that can break the flow of the conversation.

00:02:09.160 | I hope that works for you

00:02:10.600 | and doesn't hurt the listening experience.

00:02:13.180 | This show is presented by Cash App,

00:02:15.360 | the number one finance app in the App Store.

00:02:17.760 | When you get it, use code LEXPODCAST.

00:02:21.200 | Cash App lets you send money to friends,

00:02:23.520 | buy Bitcoin, and invest in the stock market

00:02:26.040 | with as little as $1.

00:02:27.920 | Brokerage services are provided by Cash App Investing,

00:02:30.960 | a subsidiary of Square, a member SIPC.

00:02:35.000 | Since Cash App allows you to send

00:02:36.720 | and receive money digitally, peer-to-peer,

00:02:39.160 | and security in all digital transactions is very important,

00:02:42.760 | let me mention the PCI Data Security Standard

00:02:45.880 | that Cash App is compliant with.

00:02:48.120 | I'm a big fan of standards for safety and security.

00:02:52.080 | PCI DSS is a good example of that,

00:02:55.080 | where a bunch of competitors got together

00:02:57.180 | and agreed that there needs to be a global standard

00:02:59.960 | around the security of transactions.

00:03:02.520 | Now, we just need to do the same for autonomous vehicles

00:03:06.040 | and AI systems in general.

00:03:08.880 | So again, if you get Cash App

00:03:10.760 | from the App Store or Google Play,

00:03:12.560 | and use the code LEXPODCAST, you'll get $10,

00:03:16.400 | and Cash App will also donate $10 to FIRST,

00:03:19.240 | one of my favorite organizations

00:03:21.280 | that is helping to advance robotics and STEM education

00:03:24.520 | for young people around the world.

00:03:26.760 | And now, here's my conversation with Markus Hutter.

00:03:31.680 | Do you think of the universe as a computer

00:03:34.480 | or maybe an information processing system?

00:03:37.020 | Let's go with a big question first.

00:03:39.080 | - Okay, with a big question first.

00:03:41.560 | I think it's a very interesting hypothesis or idea,

00:03:45.240 | and I have a background in physics,

00:03:47.960 | so I know a little bit about physical theories,

00:03:50.800 | the standard model of particle physics

00:03:52.440 | and general relativity theory,

00:03:54.440 | and they are amazing and describe

00:03:56.440 | virtually everything in the universe,

00:03:57.920 | and they're all, in a sense, computable theories.

00:03:59.760 | I mean, they're very hard to compute.

00:04:01.800 | And it's very elegant, simple theories

00:04:04.360 | which describe virtually everything in the universe.

00:04:07.260 | So there's a strong indication

00:04:10.240 | that somehow the universe is computable,

00:04:15.240 | but it's a plausible hypothesis.

00:04:17.720 | - So why do you think, just like you said,

00:04:19.520 | general relativity, quantum field theory,

00:04:22.240 | why do you think that the laws of physics

00:04:24.780 | are so nice and beautiful and simple and compressible?

00:04:28.960 | Do you think our universe was designed,

00:04:32.780 | is naturally this way?

00:04:34.200 | Are we just focusing on the parts

00:04:36.720 | that are especially compressible?

00:04:39.520 | Our human minds just enjoy something about that simplicity,

00:04:42.760 | and in fact, there's other things

00:04:44.840 | that are not so compressible.

00:04:46.720 | - No, I strongly believe, and I'm pretty convinced

00:04:49.400 | that the universe is inherently beautiful, elegant,

00:04:52.560 | and simple, and described by these equations,

00:04:55.520 | and we're not just picking that.

00:04:57.640 | I mean, if there were some phenomena

00:05:00.040 | which cannot be neatly described,

00:05:02.680 | scientists would try that, right?

00:05:04.640 | And there's biology which is more messy,

00:05:06.720 | but we understand that it's an emergent phenomena,

00:05:09.280 | and it's complex systems,

00:05:11.000 | but they still follow the same rules, right,

00:05:12.680 | of quantum and electrodynamics.

00:05:14.640 | All of chemistry follows that, and we know that.

00:05:16.560 | I mean, we cannot compute everything

00:05:18.140 | because we have limited computational resources.

00:05:20.320 | No, I think it's not a bias of the humans,

00:05:22.080 | but it's objectively simple.

00:05:24.000 | I mean, of course, you never know.

00:05:25.640 | Maybe there's some corners very far out in the universe,

00:05:28.320 | or super, super tiny below the nucleus of atoms,

00:05:33.000 | or, well, parallel universes which are not nice and simple,

00:05:38.000 | but there's no evidence for that,

00:05:40.560 | and we should apply Occam's razor

00:05:42.240 | and choose the simplest tree consistent with it,

00:05:45.160 | but also it's a little bit self-referential.

00:05:48.040 | - So maybe a quick pause.

00:05:49.480 | What is Occam's razor?

00:05:51.000 | - So Occam's razor says that you should not

00:05:53.880 | multiply entities beyond necessity,

00:05:57.480 | which sort of if you translate it to proper English means,

00:06:01.400 | and in the scientific context means

00:06:03.440 | that if you have two theories or hypotheses or models

00:06:06.440 | which equally well describe the phenomenon,

00:06:09.800 | your study or the data,

00:06:11.560 | you should choose the more simple one.

00:06:13.960 | - So that's just the principle?

00:06:15.480 | - Yes.

00:06:16.320 | - So that's not like a provable law perhaps?

00:06:20.040 | Perhaps we'll kind of discuss it and think about it,

00:06:23.480 | but what's the intuition of why the simpler answer

00:06:28.080 | is the one that is likelier to be more correct descriptor

00:06:33.080 | of whatever we're talking about?

00:06:35.080 | - I believe that Occam's razor

00:06:36.560 | is probably the most important principle in science.

00:06:40.260 | I mean, of course, we need logical deduction

00:06:42.040 | and we do experimental design,

00:06:44.520 | but science is about understanding the world,

00:06:49.520 | finding models of the world,

00:06:51.440 | and we can come up with crazy complex models

00:06:53.680 | which explain everything but predict nothing,

00:06:55.980 | but the simple model seem to have predictive power

00:07:00.200 | and it's a valid question why.

00:07:03.120 | And there are two answers to that.

00:07:05.960 | You can just accept it.

00:07:07.200 | That is the principle of science,

00:07:09.200 | and we use this principle and it seems to be successful.

00:07:12.820 | We don't know why, but it just happens to be.

00:07:15.900 | Or you can try, you know, find another principle

00:07:18.540 | which explains Occam's razor.

00:07:21.100 | And if we start with the assumption

00:07:24.120 | that the world is governed by simple rules,

00:07:27.560 | then there's a bias to our simplicity

00:07:31.400 | and applying Occam's razor

00:07:33.880 | is the mechanism to finding these rules.

00:07:37.080 | And actually in a more quantitative sense,

00:07:39.040 | and we come back to that later

00:07:40.200 | in case of somnolent deduction,

00:07:41.680 | you can rigorously prove that.

00:07:43.000 | You have to assume that the world is simple,

00:07:45.600 | then Occam's razor is the best you can do

00:07:47.680 | in a certain sense.

00:07:49.000 | - So I apologize for the romanticized question,

00:07:51.620 | but why do you think, outside of its effectiveness,

00:07:56.240 | why do you think we find simplicity

00:07:58.360 | so appealing as human beings?

00:07:59.920 | Why does it just, why does E equals MC squared

00:08:03.480 | seem so beautiful to us humans?

00:08:08.320 | - I guess mostly, in general,

00:08:10.600 | many things can be explained by an evolutionary argument.

00:08:14.960 | And, you know, there's some artifacts in humans

00:08:17.320 | which are just artifacts and not evolutionary necessary.

00:08:21.240 | But with this beauty and simplicity,

00:08:24.160 | it's, I believe, at least the core,

00:08:28.840 | is about, like science,

00:08:32.760 | finding regularities in the world,

00:08:34.520 | understanding the world,

00:08:36.000 | which is necessary for survival, right?

00:08:38.040 | If I look at a bush, right,

00:08:40.760 | and I just see noise,

00:08:42.520 | and there is a tiger, right,

00:08:43.760 | and eats me, then I'm dead.

00:08:45.000 | But if I try to find a pattern,

00:08:46.840 | and we know that humans are prone to

00:08:49.560 | find more patterns in data than they are,

00:08:54.400 | like the Mars face and all these things,

00:08:57.880 | but this bias towards finding patterns,

00:08:59.920 | even if they are non,

00:09:01.120 | but, I mean, it's best, of course, if they are,

00:09:03.880 | helps us for survival.

00:09:05.240 | - Yeah, that's fascinating.

00:09:07.440 | I haven't thought really about the,

00:09:09.880 | I thought I just loved science,

00:09:11.480 | but indeed, in terms of just for survival purposes,

00:09:16.280 | there is an evolutionary argument

00:09:18.680 | for why we find the work of Einstein so beautiful.

00:09:23.240 | Maybe a quick small tangent.

00:09:26.840 | Could you describe what Solomonov induction is?

00:09:30.680 | - Yeah, so that's a theory which I claim,

00:09:35.040 | and where Solomonov sort of claimed a long time ago

00:09:37.680 | that this solves the big philosophical problem of induction.

00:09:42.280 | And I believe the claim is essentially true.

00:09:45.080 | And what it does is the following.

00:09:47.160 | So, okay, for the picky listener,

00:09:52.040 | induction can be interpreted narrowly and widely.

00:09:55.880 | Narrow means inferring models from data.

00:09:59.000 | And widely means also then using these models

00:10:03.080 | for doing predictions,

00:10:04.040 | so predictions also part of the induction.

00:10:06.560 | So I'm a little sloppy sort of with the terminology,

00:10:09.440 | and maybe that comes from Ray Solomonov being sloppy.

00:10:13.640 | Maybe I shouldn't say that.

00:10:14.920 | (both laughing)

00:10:16.080 | He can't complain anymore.

00:10:18.160 | So let me explain a little bit this theory in simple terms.

00:10:22.160 | So assume you have a data sequence,

00:10:24.280 | make it very simple, the simplest one,

00:10:25.760 | say 1, 1, 1, 1, 1, and you see 100 1s.

00:10:28.840 | What do you think comes next?

00:10:31.000 | The natural answer, I'm gonna speed up a little bit,

00:10:32.680 | the natural answer is, of course, 1.

00:10:34.720 | And the question is why?

00:10:38.160 | Well, we see a pattern there.

00:10:41.000 | There's a 1, and we repeat it.

00:10:42.800 | And why should it suddenly after 100 1s be different?

00:10:45.520 | So what we're looking for is simple explanations or models

00:10:49.120 | for the data we have.

00:10:50.760 | And now the question is,

00:10:51.720 | a model has to be presented in a certain language.

00:10:55.480 | In which language do we use?

00:10:57.560 | In science, we want formal languages,

00:10:59.440 | and we can use mathematics,

00:11:00.840 | or we can use programs on a computer.

00:11:03.880 | So abstractly on a Turing machine, for instance,

00:11:06.440 | or it can be a general purpose computer.

00:11:08.440 | And there are, of course, lots of models.

00:11:11.400 | You can say maybe it's 100 1s,

00:11:12.880 | and then 100 0s, and 100 1s, that's a model, right?

00:11:15.280 | But there are simpler models.

00:11:17.000 | There's a model print one loop.

00:11:19.480 | It also explains the data.

00:11:21.200 | And if you push that to the extreme,

00:11:24.640 | you are looking for the shortest program,

00:11:26.880 | which, if you run this program,

00:11:28.560 | reproduces the data you have.

00:11:30.920 | It will not stop, it will continue, naturally.

00:11:33.680 | And this you take for your prediction.

00:11:36.120 | And on the sequence of 1s, it's very plausible, right?

00:11:38.440 | That print one loop is the shortest program.

00:11:40.880 | We can give some more complex examples,

00:11:43.000 | like one, two, three, four, five.

00:11:45.240 | What comes next?

00:11:46.080 | The short program is again, you know, counter.

00:11:48.200 | And so that is, roughly speaking,

00:11:51.440 | how Solomon's induction works.

00:11:53.160 | The extra twist is that it can also deal with noisy data.

00:11:57.760 | So if you have, for instance, a coin flip,

00:11:59.960 | say a biased coin, which comes up head

00:12:01.720 | with 60% probability, then it will predict.

00:12:06.320 | It will learn and figure this out.

00:12:07.800 | And after a while, it predict,

00:12:08.720 | oh, the next coin flip will be head with probability 60%.

00:12:12.640 | So it's the stochastic version of that.

00:12:14.840 | - But the goal is, the dream is,

00:12:16.720 | always the search for the short program.

00:12:18.760 | - Yes, yeah.

00:12:19.600 | Well, in Solomon of induction, precisely what you do is,

00:12:22.240 | so you combine, so looking for the shortest program

00:12:26.000 | is like applying Opus Razor,

00:12:27.640 | like looking for the simplest theory.

00:12:29.640 | There's also Epicurus principle, which says,

00:12:32.360 | if you have multiple hypothesis,

00:12:33.880 | which equally well describe your data,

00:12:35.600 | don't discard any of them,

00:12:36.760 | keep all of them around, you never know.

00:12:39.120 | And you can put it together and say,

00:12:41.000 | okay, I have a bias towards simplicity,

00:12:43.280 | but I don't rule out the larger models.

00:12:45.360 | And technically what we do is,

00:12:47.320 | we weigh the shorter models higher

00:12:50.840 | and the longer models lower.

00:12:53.040 | And you use a Bayesian techniques,

00:12:54.640 | you have a prior, which is precisely two to the minus

00:12:59.640 | the complexity of the program.

00:13:02.680 | And you weigh all this hypothesis and take this mixture

00:13:05.400 | and then you get also the stochasticity in.

00:13:07.840 | - Yeah, like many of your ideas,

00:13:09.160 | that's just a beautiful idea of weighing

00:13:11.280 | based on the simplicity of the program.

00:13:12.800 | I love that.

00:13:13.840 | That seems to me, maybe a very human-centric concept,

00:13:17.760 | seems to be a very appealing way

00:13:20.000 | of discovering good programs in this world.

00:13:25.080 | You've used the term compression quite a bit.

00:13:28.240 | I think it's a beautiful idea.

00:13:30.720 | Sort of, we just talked about simplicity

00:13:33.080 | and maybe science or just all of our intellectual pursuits

00:13:37.760 | is basically the attempt to compress the complexity

00:13:41.520 | all around us into something simple.

00:13:43.600 | So what does this word mean to you, compression?

00:13:48.600 | - I essentially have already explained it.

00:13:52.040 | So compression means for me,

00:13:54.440 | finding short programs for the data or the phenomenon at hand

00:13:59.440 | you could interpret it more widely as finding simple theories

00:14:04.440 | which can be mathematical theories

00:14:05.920 | or maybe even informal, like just in words.

00:14:09.480 | Compression means finding short descriptions,

00:14:12.360 | explanations, programs for the data.

00:14:15.320 | - Do you see science as a kind of

00:14:19.640 | our human attempt at compression?

00:14:22.120 | So we're speaking more generally

00:14:23.560 | 'cause when you say programs,

00:14:25.440 | you're kind of zooming in on a particular

00:14:26.920 | sort of almost like a computer science,

00:14:28.600 | artificial intelligence focus.

00:14:30.680 | But do you see all of human endeavor

00:14:32.400 | as a kind of compression?

00:14:34.840 | - Well, at least all of science I see as a endeavor

00:14:37.400 | of compression, not all of humanity maybe.

00:14:40.160 | And well, there are also some other aspects of science

00:14:42.640 | like experimental design, right?

00:14:44.120 | I mean, we create experiments specifically

00:14:47.960 | to get extra knowledge.

00:14:49.240 | And that is then part of the decision-making process.

00:14:52.800 | But once we have the data to understand the data

00:14:57.360 | is essentially compression.

00:14:58.640 | So I don't see any difference between compression,

00:15:01.320 | understanding and prediction.

00:15:05.060 | - So we're jumping around topics a little bit,

00:15:08.040 | but returning back to simplicity,

00:15:10.560 | a fascinating concept of Kolmogorov complexity.

00:15:14.400 | So in your sense, do most objects

00:15:17.160 | in our mathematical universe

00:15:19.720 | have high Kolmogorov complexity?

00:15:22.000 | And maybe what is, first of all,

00:15:24.120 | what is Kolmogorov complexity?

00:15:26.000 | - Okay, Kolmogorov complexity

00:15:27.840 | is a notion of simplicity or complexity.

00:15:31.200 | And it takes the compression view to the extreme.

00:15:36.000 | So I explained before that if you have some data sequence,

00:15:39.720 | just think about a file on a computer

00:15:41.760 | and best sort of, you know, just a string of bits.

00:15:45.160 | And if you, and we have data compressors,

00:15:49.440 | like we compress big files into say zip files

00:15:52.040 | with certain compressors.

00:15:53.720 | And you can also produce self-extracting archives.

00:15:56.360 | That means as an executable, if you run it,

00:15:59.120 | it reproduces your original file

00:16:00.760 | without needing an extra decompressor.

00:16:02.840 | It's just the decompressor plus the archive together in one.

00:16:06.240 | And now there are better and worse compressors.

00:16:08.800 | And you can ask, what is the ultimate compressor?

00:16:11.120 | So what is the shortest possible self-extracting archive

00:16:14.840 | you could produce for a certain data set, yeah?

00:16:17.880 | Which reproduces the data set.

00:16:19.560 | And the length of this is called the Kolmogorov complexity.

00:16:23.280 | And arguably, that is the information content

00:16:26.680 | in the data set.

00:16:27.960 | I mean, if the data set is very redundant or very boring,

00:16:30.440 | you can compress it very well.

00:16:31.720 | So the information content should be low.

00:16:34.760 | And you know, it is low according to this definition.

00:16:36.920 | - So it's the length of the shortest program

00:16:39.680 | that summarizes the data?

00:16:41.000 | - Yes, yeah.

00:16:42.040 | - And what's your sense of our sort of universe

00:16:46.280 | when we think about the different objects in our universe,

00:16:51.280 | that we try concepts or whatever at every level,

00:16:55.440 | do they have high or low Kolmogorov complexity?

00:16:58.320 | So what's the hope?

00:16:59.400 | Do we have a lot of hope

00:17:01.400 | in being able to summarize much of our world?

00:17:04.400 | - That's a tricky and difficult question.

00:17:08.520 | So as I said before, I believe that the whole universe,

00:17:13.520 | based on the evidence we have, is very simple.

00:17:16.720 | So it has a very short description.

00:17:19.240 | - Sorry, to linger on that, the whole universe,

00:17:23.200 | what does that mean?

00:17:24.040 | Do you mean at the very basic fundamental level

00:17:26.760 | in order to create the universe?

00:17:28.600 | - Yes, yeah.

00:17:29.440 | So you need a very short program,

00:17:32.120 | when you run it--

00:17:32.960 | - To get the thing going.

00:17:34.040 | - To get the thing going,

00:17:35.040 | and then it will reproduce our universe.

00:17:37.480 | There's a problem with noise.

00:17:39.360 | We can come back to that later, possibly.

00:17:42.120 | - Is noise a problem or is it a bug or a feature?

00:17:45.280 | - I would say it makes our life as a scientist

00:17:49.480 | really, really much harder.

00:17:52.200 | I mean, think about without noise,

00:17:53.520 | we wouldn't need all of the statistics.

00:17:55.980 | - But then maybe we wouldn't feel like there's a free will.

00:17:58.920 | Maybe we need that for the--

00:18:00.160 | - Yeah, this is an illusion

00:18:02.360 | that noise can give you free will.

00:18:04.640 | - At least in that way, it's a feature.

00:18:06.720 | But also, if you don't have noise,

00:18:09.080 | you have chaotic phenomena,

00:18:10.800 | which are effectively like noise.

00:18:12.800 | So we can't get away with statistics even then.

00:18:15.760 | I mean, think about rolling a dice

00:18:17.600 | and forget about quantum mechanics

00:18:19.300 | and you know exactly how you throw it.

00:18:21.260 | But I mean, it's still so hard to compute the trajectory

00:18:24.080 | that effectively it is best to model it

00:18:26.440 | as coming out with a number,

00:18:30.160 | this probability one over six.

00:18:33.120 | But from this set of philosophical

00:18:36.400 | Kolmogorov complexity perspective,

00:18:38.160 | if we didn't have noise,

00:18:39.960 | then arguably you could describe the whole universe

00:18:43.240 | as well as a standard model plus generativity.

00:18:47.460 | I mean, we don't have a theory of everything yet,

00:18:49.680 | but sort of assuming we are close to it or have it, yeah.

00:18:52.280 | Plus the initial conditions,

00:18:53.520 | which may hopefully be simple.

00:18:55.480 | And then you just run it

00:18:56.680 | and then you would reproduce the universe.

00:18:59.120 | But that's spoiled by noise or by chaotic systems

00:19:03.560 | or by initial conditions, which may be complex.

00:19:06.320 | So now if we don't take the whole universe,

00:19:09.760 | but just a subset, just take planet Earth.

00:19:13.800 | Planet Earth cannot be compressed

00:19:15.640 | into a couple of equations.

00:19:17.600 | This is a hugely complex system.

00:19:19.280 | - So interesting.

00:19:20.120 | So when you look at the window,

00:19:21.680 | like the whole thing might be simple,

00:19:23.080 | but when you just take a small window, then--

00:19:26.120 | - It may become complex and that may be counterintuitive,

00:19:28.800 | but there's a very nice analogy.

00:19:31.760 | The book, the library of all books.

00:19:34.280 | So imagine you have a normal library with interesting books

00:19:37.000 | and you go there, great, lots of information

00:19:39.360 | and quite complex, yeah?

00:19:42.040 | So now I create a library

00:19:43.360 | which contains all possible books, say, of 500 pages.

00:19:46.840 | So the first book just has AAAA over all the pages.

00:19:49.720 | The next book, AAAA and ends with B and so on.

00:19:52.280 | I create this library of all books.

00:19:54.240 | I can write a super short program

00:19:55.760 | which creates this library.

00:19:57.320 | So this library which has all books

00:19:59.040 | has zero information content.

00:20:01.320 | And you take a subset of this library

00:20:02.920 | and suddenly you have a lot of information in there.

00:20:05.360 | - So that's fascinating.

00:20:06.720 | I think one of the most beautiful object,

00:20:08.360 | mathematical objects that, at least today,

00:20:10.480 | seems to be understudied or under-talked about

00:20:12.560 | is cellular automata.

00:20:14.960 | What lessons do you draw from sort of the game of life

00:20:18.600 | for cellular automata where you start with the simple rules

00:20:20.840 | just like you're describing with the universe

00:20:22.880 | and somehow complexity emerges?

00:20:26.320 | Do you feel like you have an intuitive grasp

00:20:30.400 | on the fascinating behavior of such systems

00:20:34.120 | where, like you said, some chaotic behavior could happen,

00:20:37.560 | some complexity could emerge,

00:20:39.400 | it could die out in some very rigid structures?

00:20:43.680 | Do you have a sense about cellular automata

00:20:46.800 | that somehow transfers maybe

00:20:48.200 | to the bigger questions of our universe?

00:20:50.960 | - Yeah, the cellular automata,

00:20:52.000 | and especially the converse game of life,

00:20:54.240 | is really great because these rules are so simple.

00:20:56.240 | You can explain it to every child,

00:20:57.720 | and even by hand you can simulate a little bit,

00:21:00.280 | and you see these beautiful patterns emerge,

00:21:04.000 | and people have proven that it's even Turing-complete.

00:21:06.800 | You cannot just use a computer to simulate game of life,

00:21:09.840 | but you can also use game of life to simulate any computer.

00:21:13.480 | That is truly amazing,

00:21:16.520 | and it's the prime example probably to demonstrate

00:21:21.240 | that very simple rules can lead to very rich phenomena.

00:21:25.240 | And people sometimes,

00:21:26.800 | how is chemistry and biology so rich?

00:21:29.720 | I mean, this can't be based on simple rules,

00:21:32.400 | but no, we know quantum electrodynamics

00:21:34.520 | describes all of chemistry,

00:21:36.360 | and we come later back to that.

00:21:38.960 | I claim intelligence can be explained

00:21:40.920 | or described in one single equation,

00:21:43.000 | this very rich phenomenon.

00:21:44.600 | You asked also about whether I understand this phenomenon,

00:21:49.880 | and it's probably not,

00:21:54.280 | and there's this saying,

00:21:55.560 | you never understand really things,

00:21:56.800 | you just get used to them.

00:21:58.360 | And I think I'm pretty used to cellular automata,

00:22:03.360 | so you believe that you understand now

00:22:05.640 | why this phenomenon happens,

00:22:07.120 | but I give you a different example.

00:22:09.280 | I didn't play too much with this converse game of life,

00:22:11.800 | but a little bit more with fractals

00:22:15.040 | and with the Mandelbrot set,

00:22:16.240 | and you know, these beautiful patterns,

00:22:18.520 | just look Mandelbrot set.

00:22:20.000 | And well, when the computers were really slow

00:22:23.320 | and I just had a black and white monitor

00:22:25.360 | and programmed my own programs in assembler too.

00:22:29.120 | - Assembler, wow.

00:22:31.040 | Wow, you're legit.

00:22:32.440 | (both laughing)

00:22:33.840 | - To get these fractals on the screen,

00:22:35.760 | and it was mesmerized, and much later.

00:22:37.400 | So I returned to this, you know, every couple of years,

00:22:40.320 | and then I tried to understand what is going on,

00:22:42.880 | and you can understand a little bit.

00:22:44.880 | So I tried to derive the locations,

00:22:48.720 | you know, there are these circles and the apple shape,

00:22:53.560 | and then you have smaller Mandelbrot sets

00:22:57.360 | recursively in this set.

00:22:59.040 | And there's a way to mathematically,

00:23:01.760 | by solving high order polynomials,

00:23:03.480 | to figure out where these centers are

00:23:05.680 | and what size they are approximately.

00:23:08.080 | And by sort of mathematically approaching this problem,

00:23:12.520 | you slowly get a feeling of why things are like they are.

00:23:18.080 | And that sort of is a first step to understanding

00:23:23.080 | why this rich phenomenon appears.

00:23:24.920 | - Do you think it's possible, what's your intuition?

00:23:27.200 | Do you think it's possible to reverse engineer

00:23:28.920 | and find the short program that generated these fractals

00:23:33.680 | by looking at the fractals?

00:23:36.400 | - Well, in principle, yes.

00:23:37.800 | So, I mean, in principle, what you can do is,

00:23:42.000 | you take any data set, you take these fractals,

00:23:44.560 | or you take whatever your data set, whatever you have,

00:23:48.120 | say a picture of Conway's Game of Life,

00:23:51.000 | and you run through all programs.

00:23:53.200 | You take a program of size one, two, three, four,

00:23:55.280 | and all these programs, run them all in parallel

00:23:57.040 | in so-called dovetailing fashion,

00:23:59.080 | give them computational resources, first one 50%,

00:24:02.360 | second one half resources, and so on, and let them run.

00:24:05.520 | Wait until they hold, give an output,

00:24:07.680 | compare it to your data,

00:24:09.160 | and if some of these programs produce the correct data,

00:24:12.360 | then you stop, and then you have already some program.

00:24:14.480 | It may be a long program because it's faster.

00:24:16.880 | And then you continue,

00:24:17.920 | and you get shorter and shorter programs

00:24:19.680 | until you eventually find the shortest program.

00:24:22.480 | The interesting thing, you can never know

00:24:24.000 | whether it's the shortest program

00:24:25.520 | because there could be an even shorter program,

00:24:27.400 | which is just even slower,

00:24:29.400 | and you just have to wait, yeah?

00:24:32.160 | But asymptotically, and actually after finite time,

00:24:34.960 | you have the shortest program.

00:24:36.440 | So, this is a theoretical but completely impractical way

00:24:40.400 | of finding the underlying structure

00:24:45.920 | in every data set,

00:24:47.360 | and that is what Solomonov induction does

00:24:48.960 | and Kolmogorov complexity.

00:24:50.600 | In practice, of course, we have to approach the problem

00:24:52.600 | more intelligently, and then,

00:24:55.120 | if you take resource limitations into account,

00:25:00.680 | there's, for instance, the field of pseudo-random numbers,

00:25:03.640 | and these are random numbers,

00:25:05.800 | so these are deterministic sequences,

00:25:08.440 | but no algorithm which is fast,

00:25:10.840 | fast means runs in polynomial time,

00:25:12.560 | can detect that it's actually deterministic.

00:25:15.520 | So, we can produce interesting,

00:25:17.840 | I mean, random numbers, maybe not that interesting,

00:25:19.440 | but just an example.

00:25:20.360 | We can produce complex-looking data,

00:25:24.280 | and we can then prove that no fast algorithm

00:25:27.040 | can detect the underlying pattern.

00:25:28.920 | - Which is, unfortunately,

00:25:34.480 | that's a big challenge for our search for simple programs

00:25:39.760 | in the space of artificial intelligence, perhaps.

00:25:42.280 | - Yes, it definitely is for artificial intelligence,

00:25:44.360 | and it's quite surprising that it's,

00:25:47.120 | I can't say easy, I mean,

00:25:48.680 | physicists worked really hard to find these theories,

00:25:52.040 | but apparently, it was possible for human minds

00:25:55.920 | to find these simple rules in the universe.

00:25:57.520 | It could have been different, right?

00:25:59.080 | - It could have been different.

00:26:00.920 | It's awe-inspiring.

00:26:02.920 | So, let me ask another absurdly big question.

00:26:07.680 | What is intelligence, in your view?

00:26:13.080 | So, I have, of course, a definition.

00:26:15.160 | - I wasn't sure what you were gonna say,

00:26:18.360 | 'cause you could have just as easily said,

00:26:19.880 | "I have no clue."

00:26:21.280 | - Which many people would say,

00:26:23.600 | but I'm not modest in this question.

00:26:25.440 | So, the informal version,

00:26:30.200 | which I worked out together with Shane Lack,

00:26:33.240 | who co-founded DeepMind,

00:26:35.680 | is that intelligence measures an agent's ability

00:26:38.880 | to perform well in a wide range of environments.

00:26:43.080 | So, that doesn't sound very impressive,

00:26:46.000 | and these words have been very carefully chosen,

00:26:49.760 | and there is a mathematical theory behind that,

00:26:53.160 | and we come back to that later.

00:26:55.160 | And if you look at this definition by itself,

00:26:59.880 | it seems like, yeah, okay,

00:27:01.400 | but it seems a lot of things are missing.

00:27:03.640 | But if you think it through,

00:27:05.280 | then you realize that most,

00:27:08.960 | and I claim all of the other traits,

00:27:10.920 | at least of rational intelligence,

00:27:12.840 | which we usually associate with intelligence,

00:27:14.680 | are emergent phenomena from this definition.

00:27:18.200 | Like, you know, creativity, memorization,

00:27:20.280 | planning, knowledge.

00:27:22.400 | You all need that in order to perform well

00:27:25.160 | in a wide range of environments.

00:27:27.600 | So, you don't have to explicitly mention that

00:27:29.320 | in a definition.

00:27:30.160 | - Interesting.

00:27:31.000 | So, yeah, so the consciousness, abstract reasoning,

00:27:34.200 | all these kinds of things are just emergent phenomena

00:27:36.480 | that help you in towards,

00:27:39.880 | can you say the definition again?

00:27:42.080 | So, multiple environments.

00:27:44.400 | Did you mention the word goals?

00:27:46.080 | - No, but we have an alternative definition.

00:27:47.960 | Instead of performing well,

00:27:49.000 | you can just replace it by goals.

00:27:50.400 | So, intelligence measures an agent's ability

00:27:53.520 | to achieve goals in a wide range of environments.

00:27:55.920 | That's more or less equal.

00:27:56.760 | - But it's interesting, 'cause in there,

00:27:58.160 | there's an injection of the word goals.

00:27:59.960 | So, we wanna specify there should be a goal.

00:28:03.360 | - Yeah, but perform well is sort of,

00:28:05.000 | what does it mean?

00:28:06.440 | It's the same problem.

00:28:07.280 | - Yeah, there's a little bit of a gray area,

00:28:09.440 | but it's much closer to something that could be formalized.

00:28:12.480 | In your view, are humans,

00:28:16.520 | where do humans fit into that definition?

00:28:18.480 | Are they general intelligence systems

00:28:22.080 | that are able to perform in,

00:28:24.240 | like how good are they at fulfilling that definition,

00:28:28.000 | at performing well in multiple environments?

00:28:31.360 | - Yeah, that's a big question.

00:28:32.880 | I mean, the humans are performing best among all--

00:28:35.960 | - Species on Earth?

00:28:37.720 | - Species we know of, yeah.

00:28:40.800 | - Depends, you could say that trees and plants

00:28:43.480 | are doing a better job.

00:28:44.560 | They'll probably outlast us.

00:28:46.840 | - Yeah, but they are in a much more narrow environment.

00:28:49.520 | I mean, you just have a little bit of air pollutions

00:28:51.800 | and these trees die, and we can adapt.

00:28:54.160 | We build houses, we build filters,

00:28:55.840 | we do geoengineering.

00:28:59.600 | - So, the multiple environment part.

00:29:01.160 | - Yeah, that is very important.

00:29:02.720 | So, that distinguished narrow intelligence

00:29:04.760 | from wide intelligence, also in the AI research.

00:29:07.440 | - So, let me ask the Alan Turing question.

00:29:12.200 | Can machines think?

00:29:14.280 | Can machines be intelligent?

00:29:16.000 | So, in your view, I have to kind of ask,

00:29:19.680 | the answer's probably yes,

00:29:20.680 | but I wanna kind of hear your thoughts on it.

00:29:24.480 | Can machines be made to fulfill this definition

00:29:27.840 | of intelligence, to achieve intelligence?

00:29:30.880 | - Well, we are sort of getting there,

00:29:33.120 | and on a small scale, we are already there.

00:29:36.000 | The wide range of environments are missing,

00:29:39.120 | but we have self-driving cars,

00:29:40.440 | we have programs which play Go and chess,

00:29:42.840 | we have speech recognition.

00:29:44.560 | So, it's pretty amazing, but you can,

00:29:46.920 | these are narrow environments.

00:29:48.520 | But if you look at AlphaZero,

00:29:51.120 | that was also developed by DeepMind.

00:29:53.840 | I mean, got famous with AlphaGo,

00:29:55.480 | and then came AlphaZero a year later.

00:29:57.800 | That was truly amazing.

00:29:59.360 | So, reinforcement learning algorithm,

00:30:01.880 | which is able just by self-play to play chess,

00:30:06.120 | and then also Go.

00:30:08.680 | And I mean, yes, they're both games,

00:30:10.200 | but they're quite different games.

00:30:11.480 | And you know, there's, you didn't,

00:30:13.440 | don't feed them the rules of the game.

00:30:15.200 | And the most remarkable thing,

00:30:16.680 | which is still a mystery to me,

00:30:18.040 | that usually for any decent chess program,

00:30:21.000 | I don't know much about Go,

00:30:22.760 | you need opening books and end game tables,

00:30:25.760 | and so on, too.

00:30:26.960 | And nothing in there, nothing was put in there.

00:30:29.680 | - Especially with AlphaZero,

00:30:31.080 | the self-play mechanism, starting from scratch,

00:30:33.560 | being able to learn, actually new strategies is--

00:30:38.560 | - Yeah, it rediscovered all these famous openings

00:30:43.080 | within four hours by itself.

00:30:46.320 | What I was really happy about,

00:30:47.520 | I'm a terrible chess player, but I like Queen Gambi,

00:30:50.200 | and AlphaZero figured out that this is the best opening.

00:30:53.200 | (both laughing)

00:30:54.720 | - Finally.

00:30:56.800 | Somebody proved you correct.

00:30:58.320 | - So yes, to answer your question,

00:31:01.680 | yes, I believe that general intelligence is possible.

00:31:05.040 | And it also, I mean, it depends how you define it.

00:31:08.240 | Do you say AGI, with general intelligence,

00:31:11.480 | artificial intelligence, only refers to

00:31:14.520 | if you achieve human level or a subhuman level,

00:31:17.440 | but quite broad, is it also general intelligence?

00:31:19.960 | So we have to distinguish,

00:31:20.920 | or it's only super human intelligence,

00:31:23.320 | general artificial intelligence.

00:31:25.120 | - Is there a test in your mind, like the Turing test,

00:31:27.840 | and natural language, or some other test

00:31:29.960 | that would impress the heck out of you,

00:31:32.040 | that would kind of cross the line of

00:31:34.840 | your sense of intelligence

00:31:38.080 | within the framework that you said?

00:31:39.880 | - Well, the Turing test, well, it has been criticized a lot,

00:31:43.000 | but I think it's not as bad as some people think.

00:31:45.960 | Some people think it's too strong.

00:31:47.720 | So it tests not just for a system to be intelligent,

00:31:52.160 | but it also has to fake human--

00:31:55.160 | - Deception.

00:31:56.000 | - Deception, right, which is much harder.

00:31:59.000 | And on the other hand, they say it's too weak,

00:32:01.200 | because it just maybe fakes emotions or intelligent behavior.

00:32:06.200 | It's not real.

00:32:09.440 | But I don't think that's the problem, or a big problem.

00:32:12.000 | So if you would pass the Turing test,

00:32:14.520 | so a conversation over terminal with a bot for an hour,

00:32:20.640 | or maybe a day or so, and you can fool a human

00:32:23.360 | into not knowing whether this is a human or not,

00:32:26.160 | so that's the Turing test, I would be truly impressed.

00:32:30.280 | And we have this annual competitions, the Leupner Prize.

00:32:34.400 | And I mean, it started with Eliza,

00:32:36.000 | that was the first conversational program.

00:32:38.240 | And what is it called, the Japanese Mitsuko or so,

00:32:41.800 | that's the winner of the last couple of years.

00:32:44.720 | And-- - It's quite impressive.

00:32:45.880 | - Yeah, it's quite impressive.

00:32:46.920 | And then Google has developed Mina, right?

00:32:50.280 | Just recently, that's an open domain conversational bot,

00:32:55.240 | just a couple of weeks ago, I think.

00:32:57.600 | - Yeah, I kind of like the metric

00:32:58.800 | that sort of the Alexa Prize has proposed.

00:33:01.720 | I mean, maybe it's obvious to you, it wasn't to me,

00:33:04.040 | of setting sort of a length of a conversation.

00:33:07.760 | Like you want the bot to be sufficiently interesting

00:33:10.960 | that you'd want to keep talking to it for like 20 minutes.

00:33:13.680 | And that's a surprisingly effective in aggregate metric,

00:33:19.560 | 'cause you really, like nobody has the patience

00:33:24.560 | to be able to talk to a bot that's not interesting

00:33:27.760 | and intelligent and witty,

00:33:29.000 | and is able to go on to different tangents, jump domains,

00:33:33.000 | be able to say something interesting

00:33:35.380 | to maintain your attention.

00:33:36.720 | - And maybe many humans will also fail this test.

00:33:39.040 | - That's the, unfortunately, we set,

00:33:42.840 | just like with autonomous vehicles, with chatbots,

00:33:45.400 | we also set a bar that's way too high to reach.

00:33:48.200 | I said, the Turing test is not as bad

00:33:50.040 | as some people believe,

00:33:51.200 | but what is really not useful about the Turing test,

00:33:55.960 | it gives us no guidance how to develop these systems

00:33:59.800 | in the first place.

00:34:00.640 | Of course, we can develop them by trial and error

00:34:03.000 | and do whatever and then run the test

00:34:05.440 | and see whether it works or not.

00:34:06.880 | But a mathematical definition of intelligence

00:34:11.880 | gives us an objective, which we can then analyze

00:34:17.560 | by theoretical tools or computational,

00:34:21.640 | and maybe even prove how close we are.

00:34:25.120 | And we will come back to that later with the IXE model.

00:34:28.720 | So, I mentioned the compression, right?

00:34:31.240 | So in natural language processing,

00:34:33.280 | they achieved amazing results.

00:34:36.720 | And one way to test this, of course,

00:34:38.960 | take the system, you train it,

00:34:40.240 | and then you see how well it performs on the task.

00:34:43.200 | But a lot of performance measurement

00:34:47.520 | is done by so-called perplexity,

00:34:49.000 | which is essentially the same as complexity

00:34:51.920 | or compression length.

00:34:53.240 | So the NLP community develops new systems

00:34:55.920 | and then they measure the compression length,

00:34:57.520 | and then they have ranking and leaks

00:35:01.240 | because there's a strong correlation

00:35:02.800 | between compressing well,

00:35:04.640 | and then the system's performing well at the task at hand.

00:35:07.560 | It's not perfect, but it's good enough for them

00:35:10.720 | as an intermediate aim.

00:35:13.640 | - So you mean a measure,

00:35:16.000 | so this is kind of almost returning

00:35:18.400 | to the comical of complexity.

00:35:19.760 | So you're saying good compression

00:35:22.480 | usually means good intelligence.

00:35:24.960 | - Yes.

00:35:25.800 | - So you mentioned you're one of the only people

00:35:31.080 | who dared boldly to try to formalize

00:35:36.080 | the idea of artificial general intelligence,

00:35:38.680 | to have a mathematical framework for intelligence,

00:35:42.840 | just like as we mentioned,

00:35:44.960 | termed AIXI, A-I-X-I.

00:35:49.200 | So let me ask the basic question.

00:35:51.760 | What is AIXI?

00:35:53.400 | - Okay, so let me first say what it stands for because--

00:35:57.960 | - What it stands for, actually,

00:35:58.880 | that's probably the more basic question.

00:36:00.400 | - Yeah. (laughs)

00:36:01.640 | The first question is usually how it's pronounced,

00:36:04.400 | but finally I put it on the website how it's pronounced,

00:36:07.200 | and you figured it out.

00:36:09.080 | - Yeah.

00:36:10.560 | - The name comes from AI, artificial intelligence,

00:36:13.320 | and the X, I, is the Greek letter Xi,

00:36:16.440 | which are used for Solomonov's distribution

00:36:19.720 | for quite stupid reasons,

00:36:22.040 | which I'm not willing to repeat here in front of camera.

00:36:24.720 | (both laugh)

00:36:27.080 | So it just happened to be more or less arbitrary,

00:36:29.920 | I chose the Xi,

00:36:31.640 | but it also has nice other interpretations.

00:36:34.720 | So there are actions and perceptions in this model,

00:36:38.200 | right, an agent has actions and perceptions,

00:36:40.520 | and over time, so this is A-index-I, X-index-I,

00:36:44.680 | so there's action at time I,

00:36:46.160 | and then followed by perception at time I.

00:36:49.040 | - Yeah, we'll go with that.

00:36:50.440 | I'll edit out the first part.

00:36:51.840 | (laughs)

00:36:52.680 | I'm just kidding.

00:36:53.520 | - I have some more interpretations.

00:36:54.880 | - Yeah, good.

00:36:55.720 | - So at some point, maybe five years ago or 10 years ago,

00:36:59.320 | I discovered in Barcelona,

00:37:03.000 | it was on a big church,

00:37:04.760 | there was in stone engraved some text,

00:37:08.520 | and the word Aix appeared there a couple of times.

00:37:11.520 | (both laugh)

00:37:12.920 | I was very surprised and happy about that,

00:37:17.000 | and I looked it up, so it is Catalan language,

00:37:19.440 | and it means, with some interpretation,

00:37:21.720 | so that's it, that's the right thing to do, yeah, Eureka.

00:37:24.800 | - Oh, so it's almost like destined,

00:37:27.440 | somehow came to you in a dream.

00:37:32.080 | - And similarly, there's a Chinese word, Aixi,

00:37:34.280 | also written like Aixi,

00:37:35.400 | if you transcribe that to Pinyin,

00:37:37.480 | and the final one is that it's AI crossed with induction,

00:37:41.160 | because that is, and it's going more to the content now,

00:37:44.680 | so good old-fashioned AI is more about planning

00:37:47.400 | in known deterministic world,

00:37:48.760 | and induction is more about often,

00:37:50.640 | you know, IID data and inferring models,

00:37:53.200 | and essentially what this Aixi model does

00:37:54.880 | is combining these two.

00:37:56.160 | - And I actually also recently, I think,

00:37:59.040 | heard that in Japanese, AI means love,

00:38:02.320 | so if you can combine XI somehow with that,

00:38:06.760 | I think we can, there might be some interesting ideas there,

00:38:10.360 | so Aixi, let's then take the next step,

00:38:12.720 | can you maybe talk at the big level

00:38:16.600 | of what is this mathematical framework?

00:38:19.520 | - Yeah, so it consists essentially of two parts,

00:38:22.600 | one is the learning and induction and prediction part,

00:38:26.600 | and the other one is the planning part,

00:38:28.760 | so let's come first to the learning,

00:38:31.240 | induction, prediction part,

00:38:32.880 | which essentially I explained already before,

00:38:35.720 | so what we need for any agent to act well

00:38:40.720 | is that it can somehow predict what happens,

00:38:43.520 | I mean, if you have no idea what your actions do,

00:38:46.080 | how can you decide which actions are good or not,

00:38:48.960 | so you need to have some model of what your actions effect,

00:38:52.880 | so what you do is you have some experience,

00:38:56.200 | you build models like scientists of your experience,

00:38:59.440 | then you hope these models are roughly correct,

00:39:01.440 | and then you use these models for prediction.

00:39:03.520 | - And the model is, sorry to interrupt,

00:39:05.240 | and the model is based on your perception of the world,

00:39:08.400 | how your actions will affect that world?

00:39:10.520 | - That's not--

00:39:12.120 | - So how do you think about it?

00:39:12.960 | - That's not the important part,

00:39:14.600 | it is technically important,

00:39:16.040 | but at this stage we can just think about predicting,

00:39:18.280 | say, stock market data,

00:39:20.080 | whether data or IQ sequences,

00:39:22.240 | one, two, three, four, five, what comes next, yeah?

00:39:24.520 | So of course our actions affect what we're doing,

00:39:28.680 | but I'll come back to that in a second.

00:39:30.240 | - So, and I'll keep just interrupting,

00:39:32.160 | so just to draw a line between prediction and planning,

00:39:37.160 | what do you mean by prediction in this way?

00:39:40.880 | It's trying to predict the environment

00:39:43.640 | without your long-term action in that environment,

00:39:47.280 | what is prediction?

00:39:48.240 | - Okay, if you want to put the actions in now,

00:39:51.200 | okay, then let's put in now, yeah?

00:39:53.680 | So--

00:39:54.720 | - We don't have to put them now, scratch it,

00:39:56.840 | scratch it, dumb question, okay.

00:39:58.360 | - So the simplest form of prediction is

00:40:01.320 | that you just have data which you passively observe,

00:40:04.840 | and you want to predict what happens

00:40:06.200 | without interfering, as I said,

00:40:08.960 | weather forecasting, stock market, IQ sequences,

00:40:12.120 | or just anything, okay?

00:40:16.240 | And Solomonov's theory of induction based on compression,

00:40:18.920 | so you look for the shortest program

00:40:20.400 | which describes your data sequence,

00:40:22.280 | and then you take this program, run it,

00:40:24.440 | it reproduces your data sequence by definition,

00:40:27.000 | and then you let it continue running,

00:40:29.120 | and then it will produce some predictions,

00:40:30.880 | and you can rigorously prove that for any prediction task,

00:40:35.880 | this is essentially the best possible predictor.

00:40:40.040 | Of course, if there's a prediction task,

00:40:42.040 | or a task which is unpredictable,

00:40:45.080 | like, you know, fair coin flips,

00:40:46.720 | yeah, I cannot predict the next fair coin flip,

00:40:48.160 | what Solomonov does, he says,

00:40:49.160 | okay, next head is probably 50%,

00:40:51.640 | it's the best you can do.

00:40:52.560 | So if something is unpredictable,

00:40:54.040 | Solomonov will also not magically predict it.

00:40:56.600 | But if there is some pattern and predictability,

00:40:59.640 | then Solomonov induction will figure that out eventually,

00:41:03.720 | and not just eventually, but rather quickly,

00:41:06.040 | and you can have proof convergence rates,

00:41:08.360 | whatever your data is.

00:41:11.720 | So that is pure magic in a sense.

00:41:14.760 | What's the catch?

00:41:15.600 | Well, the catch is that it's not computable,

00:41:17.080 | and we come back to that later,

00:41:18.160 | you cannot just implement it,

00:41:19.440 | and even with Google resources here,

00:41:21.120 | and run it, and predict the stock market,

00:41:23.280 | and become rich.

00:41:24.120 | I mean, Ray Solomonov already tried it at the time.

00:41:28.120 | - But so the basic task is you're in the environment,

00:41:31.640 | and you're interacting with the environment

00:41:33.200 | to try to learn a model of that environment,

00:41:35.400 | and the model is in the space of all these programs,

00:41:38.720 | and your goal is to get a bunch of programs that are simple.

00:41:41.320 | - And so let's go to the actions now.

00:41:44.000 | But actually, good that you asked.

00:41:45.040 | Usually I skip this part,

00:41:46.360 | although there is also a minor contribution,

00:41:48.240 | which I did, so the action part,

00:41:49.680 | but I usually sort of just jump to the decision part.

00:41:51.760 | So let me explain to the action part now.

00:41:53.360 | Thanks for asking.

00:41:54.280 | So you have to modify it a little bit

00:41:57.720 | by now not just predicting a sequence

00:42:01.040 | which just comes to you,

00:42:03.200 | but you have an observation, then you act somehow,

00:42:06.720 | and then you want to predict the next observation

00:42:09.080 | based on the past observation and your action.

00:42:11.880 | Then you take the next action,

00:42:14.640 | you don't care about predicting it because you're doing it,

00:42:17.200 | and then you get the next observation,

00:42:19.000 | and you want, well, before you get it,

00:42:20.680 | you want to predict it, again,

00:42:21.840 | based on your past action and observation sequence.

00:42:24.840 | You just condition extra on your actions.

00:42:28.720 | There's an interesting alternative

00:42:30.480 | that you also try to predict your own actions.

00:42:33.360 | If you want--

00:42:36.400 | - In the past or the future?

00:42:38.000 | - Your future actions.

00:42:39.720 | - That's interesting.

00:42:40.560 | (both laughing)

00:42:41.840 | Wait, let me wrap.

00:42:43.480 | I think my brain is broke.

00:42:45.000 | - We should maybe discuss that later

00:42:47.400 | after I've explained the ICES model.

00:42:48.760 | That's an interesting variation.

00:42:50.160 | - But that is a really interesting variation.

00:42:52.080 | And a quick comment,

00:42:53.080 | I don't know if you want to insert that in here,

00:42:55.440 | but you're looking at that, in terms of observations,

00:42:59.200 | you're looking at the entire, the big history,

00:43:01.600 | the long history of the observations.

00:43:03.320 | - Exactly, that's very important,

00:43:04.440 | the whole history from birth of the agent.

00:43:07.680 | And we can come back to that also why this is important.

00:43:10.840 | Often, in RL, you have MDPs,

00:43:13.560 | micro-decision processes, which are much more limiting.

00:43:15.840 | Okay, so now we can predict conditioned on actions.

00:43:19.880 | So even if we influence environment,

00:43:21.560 | but prediction is not all we want to do, right?

00:43:24.080 | We also want to act really in the world.

00:43:26.920 | And the question is how to choose the actions.

00:43:29.080 | And we don't want to greedily choose the actions,

00:43:32.320 | just what is best in the next time step.

00:43:36.480 | And we first, I should say,

00:43:38.320 | how do we measure performance?

00:43:39.920 | So we measure performance by giving the agent reward.

00:43:43.320 | That's the so-called reinforcement learning framework.

00:43:45.600 | So every time step, you can give it a positive reward

00:43:48.520 | or negative reward, or maybe no reward.

00:43:50.320 | It could be a very scarce, right?

00:43:51.880 | Like if you play chess, just at the end of the game,

00:43:54.120 | you give plus one for winning or minus one for losing.

00:43:56.880 | So in the IXE framework, that's completely sufficient.

00:43:59.200 | So occasionally you give a reward signal

00:44:01.400 | and you ask the agent to maximize reward,

00:44:04.000 | but not greedily sort of, you know, the next one, next one,

00:44:06.360 | because that's very bad in the long run if you're greedy.

00:44:09.160 | But over the lifetime of the agent,

00:44:12.400 | so let's assume the agent lives for M time steps,

00:44:14.560 | let's just say it dies in sort of 100 years, sharp.

00:44:16.960 | That's just, you know, the simplest model to explain.

00:44:19.720 | So it looks at the future reward sum

00:44:22.320 | and ask what is my action sequence,

00:44:24.880 | or actually more precisely my policy,

00:44:26.960 | which leads in expectation,

00:44:29.880 | because I don't know the world,

00:44:32.160 | to the maximum reward sum.

00:44:34.160 | Let me give you an analogy.

00:44:36.120 | In chess, for instance,

00:44:38.240 | we know how to play optimally in theory.

00:44:40.320 | It's just a minimax strategy.

00:44:42.160 | I play the move which seems best to me

00:44:44.400 | under the assumption that the opponent plays the move

00:44:46.800 | which is best for him, so worst for me,

00:44:50.640 | under the assumption that I play, again, the best move,

00:44:54.160 | and then you have this expected max tree

00:44:55.960 | to the end of the game,

00:44:57.560 | and then you backpropagate

00:44:58.840 | and then you get the best possible move.

00:45:00.760 | So that is the optimal strategy,

00:45:02.120 | which von Neumann already figured out a long time ago,

00:45:04.840 | for playing adversarial games.

00:45:08.960 | Luckily, or maybe unluckily for the theory,

00:45:11.600 | it becomes harder.

00:45:12.440 | The world is not always adversarial,

00:45:14.960 | so it can be, if there are other humans, even cooperative,

00:45:18.400 | or nature is usually, I mean, the dead nature is stochastic.

00:45:22.680 | Things just happen randomly, or don't care about you.

00:45:26.800 | So what you have to take into account is the noise

00:45:29.400 | and not necessarily adversariality.

00:45:30.720 | So you replace the minimum on the opponent's side

00:45:34.000 | by an expectation,

00:45:36.000 | which is general enough to include also adversarial cases.

00:45:40.040 | So now instead of a minimax strategy,

00:45:41.560 | you have an expected max strategy.

00:45:43.800 | So far so good, so that is well known,

00:45:45.400 | it's called sequential decision theory.

00:45:48.000 | But the question is,

00:45:49.440 | on which probability distribution do you base that?

00:45:52.440 | If I have the true probability distribution,

00:45:55.360 | like say I play Begummin, right?

00:45:56.920 | There's dice and there's certain randomness involved.

00:45:59.320 | Yeah, I can calculate probabilities

00:46:00.920 | and feed it in the expected max

00:46:02.600 | or the sequential decision tree,

00:46:04.120 | come up with the optimal decision if I have enough compute.

00:46:07.160 | But for the real world, we don't know that.

00:46:09.720 | What is the probability the driver in front of me breaks?

00:46:13.920 | I don't know.

00:46:14.880 | So depends on all kinds of things

00:46:16.880 | and especially new situations, I don't know.

00:46:19.600 | So this is this unknown thing about prediction

00:46:22.480 | and there's where Solomonov comes in.

00:46:24.200 | So what you do is in sequential decision tree,

00:46:26.320 | you just replace the true distribution,

00:46:28.640 | which we don't know, by this universal distribution.

00:46:32.920 | I didn't explicitly talk about it,

00:46:34.600 | but this is used for universal prediction

00:46:36.760 | and plug it into the sequential decision tree mechanism.

00:46:40.400 | And then you get the best of both worlds.

00:46:42.600 | You have a long-term planning agent,

00:46:44.600 | but it doesn't need to know anything about the world

00:46:48.040 | because the Solomonov induction part learns.

00:46:51.600 | - Can you explicitly try to describe

00:46:54.680 | the universal distribution

00:46:56.040 | and how Solomonov induction plays a role here?

00:46:59.680 | I'm trying to understand.

00:47:00.760 | - So what it does it, so in the simplest case,

00:47:03.840 | I said, take the shortest program,

00:47:05.600 | describing your data, run it,

00:47:07.160 | have a prediction which would be deterministic.

00:47:09.040 | - Yes.

00:47:09.880 | - Okay, but you should not just take the shortest program,

00:47:13.160 | but also consider the longer ones,

00:47:15.320 | but keep it lower a priori probability.

00:47:18.480 | So in the Bayesian framework,

00:47:20.160 | you say a priori any distribution,

00:47:23.640 | which is a model or a stochastic program

00:47:29.360 | has a certain a priori probability,

00:47:30.760 | which is two to the minus,

00:47:32.160 | and why two to the minus length,

00:47:33.320 | you know, I could explain length of this program.

00:47:35.520 | So longer programs are punished, a priori.

00:47:39.760 | And then you multiply it

00:47:41.320 | with the so-called likelihood function,

00:47:43.840 | which is, as the name suggests,

00:47:46.680 | is how likely is this model given the data at hand.

00:47:51.000 | So if you have a very wrong model,

00:47:53.240 | it's very unlikely that this model is true,

00:47:55.000 | and so it is very small number.

00:47:56.760 | So even if the model is simple, it gets penalized by that.

00:48:00.240 | And what you do is then you take just the sum,

00:48:02.480 | or this is the average over it.

00:48:04.440 | And this gives you a probability distribution,

00:48:07.600 | so-called universal distribution,

00:48:09.200 | or Solomonov distribution.

00:48:10.520 | - So it's weighed by the simplicity of the program

00:48:13.200 | and the likelihood.

00:48:14.120 | - Yes.

00:48:15.320 | - It's kind of a nice idea.

00:48:17.280 | - Yeah.

00:48:18.120 | - So, okay.

00:48:19.640 | And then you said there's, you're planning N or M,

00:48:23.240 | or forgot the letter, steps into the future.

00:48:25.920 | So how difficult is that problem?

00:48:28.280 | What's involved there?

00:48:29.440 | - Okay, so there's- - Is it a basic

00:48:30.280 | optimization problem?

00:48:31.240 | What are we talking about?

00:48:32.080 | - So you have a planning problem up to horizon M,

00:48:34.920 | and that's exponential time in the horizon M,

00:48:38.080 | which is, I mean, it's computable, but intractable.

00:48:41.760 | I mean, even for chess, it's already intractable

00:48:43.560 | to do that exactly, and for Go.

00:48:45.440 | - But it could be also discounted kind of framework, or?

00:48:48.720 | - Yeah, so having a hard horizon, you know, at 100 years,

00:48:53.000 | it's just for simplicity of discussing the model,

00:48:55.800 | and also sometimes the math is simple.

00:48:57.680 | But there are lots of variations.

00:49:00.040 | Actually, a quite interesting parameter.

00:49:02.560 | There's nothing really problematic about it,

00:49:07.280 | but it's very interesting.

00:49:08.280 | So for instance, you think, no,

00:49:09.320 | let's let the parameter M tend to infinity, right?

00:49:12.920 | You want an agent which lives forever, right?

00:49:15.880 | If you do it naively, you have two problems.

00:49:17.520 | First, the mathematics breaks down

00:49:19.200 | because you have an infinite reward sum,

00:49:21.400 | which may give infinity,

00:49:22.760 | and getting reward 0.1 every time step is infinity,

00:49:25.600 | and giving reward one every time step is infinity,

00:49:27.640 | so equally good.

00:49:29.520 | That's not really what we want.

00:49:31.120 | Other problem is that if you have an infinite life,

00:49:35.800 | you can be lazy for as long as you want for 10 years,

00:49:38.600 | and then catch up with the same expected reward.

00:49:41.440 | And, you know, think about yourself,

00:49:44.040 | or maybe some friends or so.

00:49:47.280 | If they knew they lived forever, why work hard now?

00:49:51.480 | Just enjoy your life, and then catch up later.

00:49:54.320 | So that's another problem with infinite horizon.

00:49:56.640 | And you mentioned, yes, we can go to discounting.

00:49:59.800 | But then the standard discounting

00:50:01.200 | is so-called geometric discounting.

00:50:03.120 | So a dollar today is about worth

00:50:05.440 | as much as, you know, $1.05 tomorrow.

00:50:08.360 | So if you do this so-called geometric discounting,

00:50:10.360 | you have introduced an effective horizon.

00:50:13.000 | So the agent is now motivated

00:50:15.400 | to look ahead a certain amount of time effectively.

00:50:18.400 | It's like a moving horizon.

00:50:20.640 | And for any fixed effective horizon,

00:50:23.880 | there is a problem to solve which requires larger horizon.

00:50:28.120 | So if I look ahead, you know, five time steps,

00:50:30.520 | I'm a terrible chess player, right?

00:50:32.480 | I need to look ahead longer.

00:50:34.600 | If I play Go, I probably have to look ahead even longer.

00:50:36.800 | So for every problem, for every horizon,

00:50:40.280 | there is a problem which this horizon cannot solve.

00:50:43.800 | But I introduced the so-called near harmonic horizon,

00:50:46.960 | which goes down with one over T,

00:50:48.360 | rather than exponentially T, which produces an agent

00:50:51.600 | which effectively looks into the future

00:50:53.880 | proportional to its age.

00:50:55.200 | So if it's five years old, it plans for five years.

00:50:57.360 | If it's 100 years old, it then plans for 100 years.

00:51:00.440 | And it's a little bit similar to humans too, right?

00:51:02.480 | I mean, children don't plan ahead very long,

00:51:04.320 | but then we get adult, we play ahead more longer.

00:51:07.040 | Maybe when we get very old,

00:51:08.560 | I mean, we know that we don't live forever,

00:51:10.320 | maybe then our horizon shrinks again.

00:51:12.560 | - So that's really interesting.

00:51:16.040 | So adjusting the horizon,

00:51:18.120 | is there some mathematical benefit of that?

00:51:20.680 | Or is it just a nice, I mean, intuitively,

00:51:24.800 | empirically, it would probably be a good idea

00:51:26.560 | to sort of push the horizon back,

00:51:27.960 | extend the horizon as you experience more of the world.

00:51:32.960 | But is there some mathematical conclusions here

00:51:35.880 | that are beneficial?

00:51:37.280 | - With the solomon-hawking sort of prediction part,

00:51:38.960 | we have extremely strong finite time,

00:51:41.440 | but finite data results.

00:51:44.760 | So you have so and so much data,

00:51:46.000 | then you lose so and so much.

00:51:47.160 | So the theory is really great.

00:51:49.400 | With the Ixc model, with the planning part,

00:51:51.920 | many results are only asymptotic,

00:51:54.600 | which, well, this is--

00:51:56.880 | - What does asymptotic mean?

00:51:57.720 | - Asymptotic means you can prove, for instance,

00:51:59.960 | that in the long run, if the agent acts long enough,

00:52:03.440 | then it performs optimal or some nice thing happens.

00:52:06.840 | But you don't know how fast it converges.

00:52:09.520 | So it may converge fast,

00:52:10.920 | but we're just not able to prove it

00:52:12.320 | because of a difficult problem.

00:52:13.800 | Or maybe there's a bug in the model

00:52:17.360 | so that it's really that slow.

00:52:19.560 | So that is what asymptotic means, sort of eventually,

00:52:22.880 | but we don't know how fast.

00:52:24.720 | And if I give the agent a fixed horizon M,

00:52:28.000 | then I cannot prove asymptotic results, right?

00:52:32.280 | So I mean, sort of if it dies in 100 years,

00:52:35.040 | then 100 years is over, I cannot say eventually.

00:52:37.880 | So this is the advantage of the discounting

00:52:40.640 | that I can prove asymptotic results.

00:52:42.800 | - So just to clarify, so, okay, I've built up a model.

00:52:49.160 | Well, now in the moment,

00:52:50.960 | I have this way of looking several steps ahead.

00:52:55.400 | How do I pick what action I will take?

00:52:57.840 | - It's like with a playing chess, right?

00:53:00.760 | You do this minimax.

00:53:02.360 | In this case here, do expectimax

00:53:04.440 | based on the Solomonov distribution.

00:53:06.880 | You propagate back.

00:53:08.040 | And then, well, an action falls out.

00:53:12.120 | The action which maximizes the future expected reward

00:53:15.520 | on the Solomonov distribution,

00:53:16.800 | and then you just take this action.

00:53:18.280 | - And then repeat.

00:53:19.640 | - And then you get a new observation,

00:53:21.000 | and you feed it in this action observation,

00:53:22.640 | then you repeat.

00:53:23.480 | - And the reward, so on.

00:53:24.920 | - Yeah, so you're in a row too, yeah.

00:53:26.800 | - And then maybe you can even predict your own action.

00:53:29.120 | I love that idea.

00:53:30.000 | But okay, this big framework, what is it?

00:53:34.120 | I mean, it's kind of a beautiful mathematical framework

00:53:38.920 | to think about artificial general intelligence.

00:53:41.920 | What can you, what does it help you into it

00:53:45.840 | about how to build such systems?

00:53:49.120 | Or maybe from another perspective,

00:53:51.760 | what does it help us in understanding AGI?

00:53:56.760 | - So when I started in the field,

00:54:00.520 | I was always interested in two things.

00:54:01.880 | One was AGI, the name didn't exist then,

00:54:05.880 | but what called general AI or strong AI,

00:54:09.280 | and the physics here of everything.

00:54:10.880 | So I switched back and forth between computer science

00:54:13.200 | and physics quite often.

00:54:14.760 | - You said the theory of everything.

00:54:16.000 | - The theory of everything, yeah, just like--

00:54:17.400 | - Those are basically the two biggest problems

00:54:19.280 | before all of humanity.

00:54:21.440 | - Yeah, I can explain if you wanted some later time,

00:54:28.400 | why I'm interested in these two questions.

00:54:29.960 | - Can I ask you, on a small tangent,

00:54:32.120 | if it was one to be solved, which one would you,

00:54:37.960 | if an apple fell on your head,

00:54:41.800 | and there was a brilliant insight,

00:54:43.280 | and you could arrive at the solution to one,

00:54:46.360 | would it be AGI or the theory of everything?

00:54:49.160 | - Definitely AGI, because once the AGI problem is solved,

00:54:51.760 | I can ask the AGI to solve the other problem for me.

00:54:54.400 | - Yeah, brilliantly put.

00:54:57.720 | Okay, so as you were saying about it--

00:55:01.160 | - Okay, so, and the reason why I didn't settle,

00:55:04.920 | I mean, this thought about, once you have solved AGI,

00:55:08.400 | it solves all kinds of other,

00:55:09.960 | not just the theory of every problem,

00:55:11.240 | but all kinds of more useful problems to humanity

00:55:14.200 | is very appealing to many people,

00:55:16.320 | and I had this thought also,

00:55:18.280 | but I was quite disappointed with the state of the art

00:55:23.280 | of the field of AI.

00:55:25.480 | There was some theory about logical reasoning,

00:55:28.160 | but I was never convinced that this will fly,

00:55:30.640 | and then there was this more heuristic approaches

00:55:33.360 | with neural networks, and I didn't like these heuristics,

00:55:37.520 | so, and also I didn't have any good idea myself.

00:55:40.880 | (laughing)

00:55:42.200 | So, that's the reason why I toggled back and forth

00:55:44.280 | quite some while, and even worked four and a half years

00:55:46.400 | in a company developing software,

00:55:48.280 | something completely unrelated,

00:55:49.720 | but then I had this idea about the AXI model,

00:55:52.840 | and so what it gives you, it gives you a gold standard.

00:55:57.800 | So, I have proven that this is the most intelligent agents

00:56:02.400 | which anybody could "build" in quotation mark,

00:56:06.840 | because it's just mathematical,

00:56:08.240 | and you need infinite compute, yeah?

00:56:11.200 | But this is the limit, and this is completely specified.

00:56:14.960 | It's not just a framework, and every year,

00:56:19.320 | tens of frameworks are developed,

00:56:21.240 | which are just skeletons, and then pieces are missing,

00:56:23.960 | and usually these missing pieces turn out

00:56:25.760 | to be really, really difficult,

00:56:27.400 | and so this is completely and uniquely defined,

00:56:31.160 | and we can analyze that mathematically,

00:56:33.520 | and we have also developed some approximations.

00:56:37.360 | I can talk about that a little bit later.

00:56:40.320 | That would be sort of the top-down approach,

00:56:41.840 | like, say, for Neumann's minimax theory,

00:56:44.280 | that's the theoretical optimal play of games,

00:56:47.320 | and now we need to approximate it,

00:56:48.880 | put heuristics in, prune the tree, blah, blah, blah,

00:56:51.080 | and so on, so we can do that also with the AXI model,

00:56:53.200 | but for general AI.

00:56:54.320 | It can also inspire those, and most of,

00:56:59.000 | most researchers go bottom-up, right?

00:57:00.880 | They have their systems, they try to make it more general,

00:57:02.920 | more intelligent.

00:57:04.200 | It can inspire in which direction to go.

00:57:07.760 | - What do you mean by that?

00:57:09.160 | - So if you have some choice to make, right?

00:57:11.240 | So how should I evaluate my system

00:57:13.160 | if I can't do cross-validation?

00:57:15.440 | How should I do my learning

00:57:18.080 | if my standard regularization doesn't work well, yeah?

00:57:21.560 | So the answer is always this.

00:57:22.560 | We have a system which does everything that's AXI.

00:57:25.080 | It's just, you know, completely in the ivory tower,

00:57:27.840 | completely useless from a practical point of view,

00:57:30.640 | but you can look at it and see,

00:57:31.960 | ah, yeah, maybe, you know, I can take some aspects,

00:57:34.960 | and, you know, instead of Kolmogorov complexity,

00:57:36.560 | I just take some compressors

00:57:38.120 | which has been developed so far.

00:57:39.920 | And for the planning, well, we have UCT,

00:57:42.120 | which has also, you know, been used in Go,

00:57:45.240 | and at least it's inspired me a lot

00:57:50.080 | to have this formal definition.

00:57:54.200 | And if you look at other fields, you know,

00:57:55.800 | like I always come back to physics

00:57:57.760 | because I have a physics background.

00:57:59.000 | Think about the phenomenon of energy.

00:58:00.720 | That was a long time a mysterious concept,

00:58:03.200 | and at some point it was completely formalized.

00:58:05.880 | And that really helped a lot.

00:58:08.160 | And I can point out a lot of these things

00:58:10.720 | which were first mysterious and vague,

00:58:12.960 | and then they have been rigorously formalized.

00:58:15.160 | Speed and acceleration has been confused, right,

00:58:18.240 | until it was formally defined.

00:58:19.680 | Yeah, there was a time like this.

00:58:20.800 | And people, you know, often, you know,

00:58:23.240 | who don't have any background, you know, still confuse it.

00:58:26.200 | So, and this AXI model or the intelligence definitions,

00:58:31.920 | which is sort of the dual to it,

00:58:33.160 | we come back to that later,

00:58:34.680 | formalizes the notion of intelligence

00:58:37.200 | uniquely and rigorously.

00:58:38.920 | - So in a sense, it serves as kind of the light

00:58:41.640 | at the end of the tunnel.

00:58:43.040 | - Yes, yeah.

00:58:43.880 | - So for, so, I mean, there's a million questions

00:58:46.840 | I could ask her.

00:58:47.760 | So maybe kind of, okay,

00:58:50.320 | let's feel around in the dark a little bit.

00:58:52.120 | So there's been here at DeepMind,

00:58:54.760 | but in general, been a lot of breakthrough ideas,

00:58:56.980 | just like we've been saying around reinforcement learning.

00:58:59.500 | So how do you see the progress

00:59:02.120 | in reinforcement learning is different?

00:59:04.440 | Like which subset of AXI does it occupy?

00:59:08.080 | The current, like you said,

00:59:10.600 | maybe the Markov assumption is made quite often

00:59:14.520 | in reinforcement learning.

00:59:15.880 | There's other assumptions made

00:59:20.240 | in order to make the system work.

00:59:21.560 | What do you see as the difference connection

00:59:24.200 | between reinforcement learning and AXI?

00:59:26.800 | - Yeah, so the major difference is that

00:59:29.000 | essentially all other approaches,

00:59:33.320 | they make stronger assumptions.

00:59:35.640 | So in reinforcement learning,

00:59:36.720 | the Markov assumption is that the next state

00:59:40.800 | or next observation only depends

00:59:42.040 | on the previous observation and not the whole history,

00:59:45.280 | which makes, of course, the mathematics much easier

00:59:47.560 | rather than dealing with histories.

00:59:49.840 | Of course, they profit from it also

00:59:51.640 | because then you have algorithms

00:59:53.120 | that run on current computers

00:59:54.360 | and do something practically useful.

00:59:56.640 | But for general AI, all the assumptions

00:59:59.680 | which are made by other approaches,

01:00:01.760 | we know already now they are limiting.

01:00:04.040 | So for instance, usually you need an ergodicity assumption

01:00:09.600 | in the MDP framework in order to learn.

01:00:11.680 | Ergodicity essentially means that you can recover

01:00:14.760 | from your mistakes and that there are no traps

01:00:16.720 | in the environment.

01:00:18.280 | And if you make this assumption,

01:00:20.000 | then essentially you can go back to a previous state,

01:00:22.960 | go there a couple of times,

01:00:24.160 | and then learn what statistics and what the state is like.

01:00:29.960 | And then in the long run, perform well in this state.

01:00:33.320 | But there are no fundamental problems.

01:00:35.880 | But in real life, we know there can be one single action.

01:00:39.120 | One second of being inattentive while driving a car fast

01:00:44.120 | can ruin the rest of my life.

01:00:45.960 | I can become quadriplegic or whatever.

01:00:48.400 | So there's no recovery anymore.

01:00:50.360 | So the real world is not ergodic, I always say.

01:00:52.800 | There are traps and there are situations

01:00:54.520 | where you're not recover from.

01:00:56.440 | And very little theory has been developed for this case.

01:01:01.440 | - What about, what do you see in the context of Aixia

01:01:07.280 | as the role of exploration?

01:01:09.680 | Sort of, you mentioned in the real world

01:01:14.680 | we can get into trouble when we make the wrong decisions

01:01:17.880 | and really pay for it.

01:01:19.240 | But exploration seems to be fundamentally important

01:01:22.200 | for learning about this world, for gaining new knowledge.

01:01:25.560 | So is exploration baked in?

01:01:29.120 | Another way to ask it, what are the parameters

01:01:32.240 | of Aixia that can be controlled?

01:01:35.360 | - Yeah, I say the good thing is that

01:01:38.000 | there are no parameters to control.

01:01:40.200 | Some other people try knobs to control,

01:01:43.120 | and you can do that.

01:01:44.120 | I mean, you can modify Aixia so that you have some knobs

01:01:46.880 | to play with if you want to.

01:01:48.800 | But the exploration is directly baked in.

01:01:53.640 | And that comes from the Bayesian learning

01:01:56.960 | and the long-term planning.

01:01:58.680 | So these together already imply exploration.

01:02:03.680 | You can nicely and explicitly prove that

01:02:08.240 | for simple problems like so-called bandit problems,

01:02:13.240 | where you say, to give a real world example,

01:02:18.000 | say you have two medical treatments, A and B,

01:02:20.200 | you don't know the effectiveness,

01:02:21.560 | you try A a little bit, B a little bit,

01:02:23.360 | but you don't want to harm too many patients.

01:02:25.760 | So you have to sort of trade off exploring.

01:02:29.760 | And at some point you want to explore,

01:02:31.680 | and you can do the mathematics

01:02:34.040 | and figure out the optimal strategy.

01:02:36.040 | The so-called Bayesian agents,

01:02:39.080 | they're also non-Bayesian agents,

01:02:41.080 | but it shows that this Bayesian framework,

01:02:44.200 | by taking a prior over possible worlds,

01:02:47.360 | doing the Bayesian mixture,

01:02:48.400 | then the Bayes optimal decision with long-term planning

01:02:50.600 | that is important, automatically implies exploration

01:02:55.600 | also to the proper extent,

01:02:57.600 | not too much exploration and not too little,

01:02:59.680 | in these very simple settings.

01:03:01.520 | In the IXE model, I was also able to prove

01:03:04.360 | that it is a self-optimizing theorem

01:03:06.160 | or asymptotic optimality theorems,

01:03:07.720 | although they're only asymptotic, not finite time bounds.

01:03:10.480 | - So it seems like the long-term planning

01:03:12.240 | is really important, but the long-term part

01:03:14.080 | of the planning is really important.

01:03:15.400 | - Yes. - And also,

01:03:17.320 | maybe a quick tangent,

01:03:18.920 | how important do you think is removing

01:03:21.360 | the Markov assumption and looking at the full history?

01:03:25.320 | Sort of intuitively, of course, it's important,

01:03:28.040 | but is it like fundamentally transformative

01:03:30.960 | to the entirety of the problem?

01:03:33.400 | What's your sense of it?

01:03:34.560 | 'Cause we make that assumption quite often,

01:03:37.400 | just throwing away the past.

01:03:40.000 | - No, I think it's absolutely crucial.

01:03:41.880 | The question is whether there's a way to deal with it

01:03:47.240 | in a more heuristic and still sufficiently well way.

01:03:52.240 | So I have to come up with an example on the fly,

01:03:55.520 | but you have some key event in your life,

01:03:59.400 | long time ago, in some city or something,

01:04:02.080 | you realize that's a really dangerous street or whatever,

01:04:04.680 | right, yeah, and you want to remember that forever, right,

01:04:08.000 | in case you come back there.

01:04:09.800 | - Kind of a selective kind of memory.

01:04:11.560 | So you remember all the important events in the past,

01:04:15.160 | but somehow selecting the importance is--

01:04:17.480 | - That's very hard, yeah, and I'm not concerned

01:04:19.960 | about just storing the whole history,

01:04:21.720 | just you can calculate human life,

01:04:24.720 | say, 30 or 100 years, doesn't matter, right,

01:04:27.640 | how much data comes in through the vision system

01:04:31.800 | and the auditory system, you compress it a little bit,

01:04:35.200 | in this case, lossily, and store it.

01:04:37.560 | We are soon in the means of just storing it,

01:04:40.520 | but you still need to do selection for the planning part

01:04:44.920 | and the compression for the understanding part.

01:04:47.280 | The raw storage, I'm really not concerned about,

01:04:50.000 | and I think we should just store, if you develop an agent,

01:04:53.640 | preferably just store all the interaction history,

01:04:59.400 | and then you build, of course, models on top of it,

01:05:02.240 | and you compress it, and you are selective,

01:05:04.960 | but occasionally, you go back to the old data

01:05:08.120 | and reanalyze it based on your new experience you have.

01:05:11.840 | You know, sometimes you are in school,

01:05:13.840 | you learn all these things you think is totally useless,

01:05:16.800 | and much later, you realize, oh, they were not

01:05:19.320 | so useless as you thought.

01:05:21.600 | - I'm looking at you, linear algebra.

01:05:24.080 | Right, so maybe let me ask about objective functions,

01:05:27.720 | because that reward, it seems to be an important part.

01:05:32.720 | The rewards are kind of given to the system.

01:05:36.660 | For a lot of people, the specification

01:05:44.040 | of the objective function is a key part of intelligence,

01:05:48.080 | like the agent itself figuring out what is important.

01:05:52.920 | What do you think about that?

01:05:54.640 | Is it possible within the IXE framework

01:05:58.560 | to yourself discover the reward

01:06:01.840 | based on which you should operate?

01:06:03.700 | - Okay, that will be a long answer.

01:06:07.200 | (Lex laughs)

01:06:08.760 | So, and that is a very interesting question,

01:06:11.880 | and I'm asked a lot about this question.

01:06:14.560 | Where do the rewards come from?

01:06:16.680 | And that depends, yeah?

01:06:19.200 | So, and then, you know, I give you now a couple of answers.

01:06:22.480 | So if we want to build agents, now let's start simple.

01:06:27.480 | So let's assume we want to build an agent

01:06:29.920 | based on the IXE model, which performs a particular task.

01:06:34.400 | Let's start with something super simple,

01:06:35.800 | like, I mean, super simple, like playing chess,

01:06:38.040 | yeah, or go or something, yeah?

01:06:39.880 | Then you just, you know, the reward is, you know,

01:06:42.360 | winning the game is plus one,

01:06:43.520 | losing the game is minus one, done.

01:06:46.200 | You apply this agent.

01:06:47.360 | If you have enough compute, you let itself play,

01:06:50.120 | and it will learn the rules of the game,

01:06:51.800 | will play perfect chess.

01:06:53.040 | After some while, problem solved, okay?

01:06:56.040 | So if you have more complicated problems,

01:06:59.520 | then you may believe that you have the right reward,

01:07:04.560 | but it's not.

01:07:05.400 | So a nice, cute example is elevator control

01:07:09.000 | that is also in Rich Sutton's book,

01:07:10.960 | which is a great book, by the way.

01:07:12.640 | So you control the elevator, and you think,

01:07:16.200 | well, maybe the reward should be coupled

01:07:18.320 | to how long people wait in front of the elevator.

01:07:20.440 | You know, long wait is bad.

01:07:22.360 | You program it, and you do it,

01:07:24.240 | and what happens is the elevator eagerly picks up

01:07:26.360 | all the people, but never drops them off.

01:07:28.800 | (both laughing)

01:07:30.960 | So then you realize, ah, maybe the time

01:07:33.040 | in the elevator also counts, so you minimize the sum, yeah?

01:07:36.800 | And the elevator does that, but never picks up the people

01:07:39.520 | in the 10th floor and the top floor,

01:07:40.920 | because in expectation, it's not worth it.

01:07:42.840 | Just let them stay.

01:07:43.840 | - Yeah. (both laughing)

01:07:46.360 | - So even in apparently simple problems,

01:07:50.160 | you can make mistakes, yeah?

01:07:51.760 | And that's what, in more serious context,

01:07:55.720 | say, AGI safety researchers consider.

01:07:58.520 | So now let's go back to general agents.

01:08:01.120 | So assume you want to build an agent

01:08:02.880 | which is generally useful to humans, yeah?

01:08:05.640 | So you have a household robot, yeah?

01:08:08.000 | And it should do all kinds of tasks.

01:08:10.440 | So in this case, the human should give the reward on the fly.

01:08:15.040 | I mean, maybe it's pre-trained in the factory

01:08:16.800 | and that there's some sort of internal reward

01:08:18.600 | for the battery level or whatever, yeah?

01:08:20.520 | But, so it does the dishes badly.

01:08:23.600 | You know, you punish the robot, it does it good.

01:08:25.280 | You reward the robot, and then train it to a new task,

01:08:28.000 | kind of like a child, right?

01:08:29.000 | So you need the human in the loop

01:08:31.720 | if you want a system which is useful to the human.

01:08:35.080 | And as long as this agent stays sub-human level,

01:08:38.520 | that should work reasonably well,

01:08:41.560 | apart from, you know, these examples.

01:08:43.560 | It becomes critical if they become, you know,

01:08:45.680 | on a human level.

01:08:46.520 | It's the same as children, small children.

01:08:47.800 | You have reasonably well under control.

01:08:49.360 | They become older.

01:08:51.160 | The reward technique doesn't work so well anymore.

01:08:53.720 | So then finally, so this would be agents

01:08:59.160 | which are just, you could say, slaves to the humans, yeah?

01:09:02.400 | So if you are more ambitious and just say,

01:09:04.560 | we want to build a new species of intelligent beings,

01:09:08.640 | we put them on a new planet,

01:09:09.960 | and we want them to develop this planet or whatever.

01:09:12.680 | So we don't give them any reward.

01:09:15.920 | So what could we do?

01:09:17.480 | And you could try to, you know,

01:09:20.280 | come up with some reward functions like, you know,

01:09:22.240 | it should maintain itself, the robot.

01:09:23.960 | It should maybe multiply, build more robots, right?

01:09:28.720 | And, you know, maybe, well, all kinds of things

01:09:32.320 | which you find useful, but that's pretty hard, right?

01:09:34.280 | You know, what does self-maintenance mean?

01:09:36.440 | You know, what does it mean to build a copy?

01:09:38.160 | Should it be exact copy, an approximate copy?

01:09:40.720 | And so that's really hard.

01:09:42.080 | But Laurent Assor, also at DeepMind,

01:09:46.200 | developed a beautiful model.

01:09:48.840 | So he just took the EICSI model

01:09:50.600 | and coupled the rewards to information gain.

01:09:55.040 | So he said the reward is proportional

01:09:57.920 | to how much the agent had learned about the world.

01:10:00.760 | And you can rigorously, formally, uniquely define that

01:10:03.360 | in terms of our Kettler versions, okay?

01:10:05.880 | So if you put that in,

01:10:07.400 | you get a completely autonomous agent.

01:10:09.880 | And actually, interestingly, for this agent,

01:10:11.720 | we can prove much stronger result

01:10:13.160 | than for the general agent, which is also nice.

01:10:16.000 | And if you let this agent loose,

01:10:18.120 | it will be, in a sense, the optimal scientist.

01:10:20.040 | It is absolutely curious to learn

01:10:21.680 | as much as possible about the world.

01:10:24.120 | And of course, it will also have

01:10:25.760 | a lot of instrumental goals, right?

01:10:27.200 | In order to learn, it needs to at least survive, right?

01:10:29.600 | A dead agent is not good for anything.

01:10:31.560 | So it needs to have self-preservation.

01:10:34.000 | And if it builds small helpers,

01:10:35.840 | acquiring more information, it will do that, yeah?

01:10:39.160 | If exploration, space exploration or whatever is necessary,

01:10:43.720 | right, to gathering information and develop it.

01:10:45.960 | So it has a lot of instrumental goals

01:10:48.240 | following on this information gain.

01:10:51.040 | And this agent is completely autonomous of us.

01:10:53.800 | No rewards necessary anymore.

01:10:55.680 | - Yeah, of course, it could find a way

01:10:57.560 | to game the concept of information

01:10:59.760 | and get stuck in that library that you mentioned beforehand

01:11:04.760 | with a very large number of books.

01:11:08.680 | - The first agent had this problem.

01:11:10.760 | It would get stuck in front of an old TV screen,

01:11:13.720 | which has just had white noise.

01:11:15.040 | - Yeah, white noise, yeah.

01:11:16.560 | - But the second version can deal with at least stochasticity.

01:11:20.680 | Well. - Yeah.

01:11:22.560 | What about curiosity, this kind of word,

01:11:25.440 | curiosity, creativity?

01:11:28.000 | Is that kind of the reward function being

01:11:30.960 | of getting new information,

01:11:32.000 | is that similar to idea of kind of injecting exploration

01:11:37.000 | for its own sake inside the reward function?

01:11:41.960 | Do you find this at all appealing, interesting?

01:11:44.960 | - I think that's a nice definition.

01:11:46.400 | Curiosity is reward, sorry,

01:11:49.160 | curiosity is exploration for its own sake.

01:11:51.960 | Yeah, I would accept that.

01:11:57.200 | But most curiosity, well, in humans,

01:12:00.000 | and especially in children,

01:12:01.320 | is not just for its own sake,

01:12:03.120 | but for actually learning about the environment

01:12:06.040 | and for behaving better.

01:12:08.520 | So I think most curiosity is tied, in the end,

01:12:13.200 | towards performing better.

01:12:14.920 | - Well, okay, so if intelligent systems

01:12:17.720 | need to have this reward function,

01:12:19.480 | you're an intelligent system,

01:12:22.480 | currently passing the Turing test quite effectively.

01:12:26.160 | (Markus laughs)

01:12:27.840 | What's the reward function

01:12:30.280 | of our human intelligence existence?

01:12:33.960 | What's the reward function

01:12:35.200 | that Markus Hodder is operating under?

01:12:37.760 | - Okay, to the first question,

01:12:39.800 | the biological reward function is to survive and to spread,

01:12:44.520 | and very few humans are able to overcome

01:12:48.280 | this biological reward function.

01:12:50.000 | But we live in a very nice world

01:12:54.240 | where we have lots of spare time

01:12:56.280 | and can still survive and spread,

01:12:57.680 | so we can develop arbitrary other interests,

01:13:01.960 | which is quite interesting.

01:13:03.360 | - On top of that.

01:13:04.440 | - On top of that, yeah.

01:13:06.240 | But the survival and spreading is, I would say,

01:13:09.720 | the goal or the reward function of humans, the core one.

01:13:14.400 | - I like how you avoided answering the second question,

01:13:17.480 | which a good intelligent system would.

01:13:19.760 | - So my--

01:13:20.920 | - Your own meaning of life and the reward function.

01:13:24.200 | - My own meaning of life and reward function

01:13:26.960 | is to find an AGI to build it.

01:13:29.560 | (Markus laughs)

01:13:31.200 | - Beautifully put, okay.

01:13:32.620 | Let's dissect the X even further.

01:13:34.280 | So one of the assumptions is,

01:13:36.960 | kind of, infinity keeps creeping up everywhere.

01:13:39.680 | (Markus laughs)

01:13:42.600 | Which, what are your thoughts

01:13:44.920 | on kind of bounded rationality

01:13:46.920 | and sort of the nature of our existence

01:13:50.040 | in intelligent systems is that we're operating

01:13:52.000 | always under constraints, under limited time,

01:13:55.640 | limited resources.

01:13:57.640 | How does that, how do you think about that

01:13:59.480 | within the IXE framework,

01:14:01.600 | within trying to create an AGI system

01:14:04.480 | that operates under these constraints?

01:14:06.760 | - Yeah, that is one of the criticisms about IXE,

01:14:09.200 | that it ignores computation completely,

01:14:11.320 | and some people believe that intelligence

01:14:13.800 | is inherently tied to what's bounded resources.

01:14:18.800 | - What do you think on this one point?

01:14:21.120 | Do you think it's, do you think the bounded resources

01:14:23.920 | are fundamental to intelligence?

01:14:25.640 | - I would say that an intelligence notion

01:14:31.160 | which ignores computational limits is extremely useful.

01:14:35.520 | A good intelligence notion which includes these resources

01:14:39.160 | would be even more useful, but we don't have that yet.

01:14:42.140 | And so look at other fields outside of computer science.

01:14:48.480 | Computational aspects never play a fundamental role.

01:14:52.240 | You develop biological models for cells,

01:14:54.880 | something in physics, these theories,

01:14:56.680 | I mean, become more and more crazy

01:14:58.160 | and harder and harder to compute.

01:15:00.320 | Well, in the end, of course, we need to do something

01:15:02.400 | with this model, but there's more nuisance than a feature.

01:15:05.520 | And I'm sometimes wondering if artificial intelligence

01:15:10.040 | would not sit in a computer science department,

01:15:12.080 | but in a philosophy department,

01:15:14.040 | then this computational focus

01:15:16.120 | would be probably significantly less.

01:15:18.400 | I mean, think about the induction problem

01:15:19.720 | is more in the philosophy department.

01:15:22.080 | There's virtually no paper who cares about, you know,

01:15:24.480 | how long it takes to compute the answer.

01:15:26.440 | That is completely secondary.

01:15:28.320 | Of course, once we have figured out the first problem,

01:15:31.660 | so intelligence without computational resources,

01:15:35.800 | then the next and very good question is,

01:15:39.360 | could we improve it by including computational resources?

01:15:42.420 | But nobody was able to do that so far

01:15:45.480 | in an even halfway satisfactory manner.

01:15:48.780 | - I like that, that in the long run,

01:15:51.580 | the right department to belong to is philosophy.

01:15:53.980 | That's actually quite a deep idea of,

01:15:59.580 | or even to at least to think about big picture

01:16:01.940 | philosophical questions, big picture questions,

01:16:05.340 | even in the computer science department.

01:16:07.420 | But you've mentioned approximation,

01:16:10.060 | sort of there's a lot of infinity,

01:16:12.220 | a lot of huge resources needed.

01:16:13.940 | Are there approximations to IEC

01:16:16.340 | that within the IEC framework that are useful?

01:16:19.820 | - Yeah, we have developed a couple of approximations.

01:16:23.140 | And what we do there is that the Solomoff induction part,

01:16:28.140 | which was, you know, find the shortest program

01:16:32.440 | describing your data,

01:16:33.680 | which has replaced it by standard data compressors, right?

01:16:36.700 | And the better compressors get, you know,

01:16:39.280 | the better this part will become.

01:16:41.740 | We focus on a particular compressor

01:16:43.420 | called context-free weighting,

01:16:44.580 | which is pretty amazing, not so well known.

01:16:48.540 | And has beautiful theoretical properties,

01:16:50.140 | also works reasonably well in practice.

01:16:52.260 | So we use that for the approximation of the induction

01:16:55.180 | and the learning and the prediction part.

01:16:57.220 | And for the planning part,

01:17:01.740 | we essentially just took the ideas

01:17:03.740 | from a computer go from 2006.

01:17:07.340 | It was Java Zipispari, also now at DeepMind,

01:17:11.340 | who developed the so-called UCT algorithm,

01:17:14.620 | upper confidence bound for trees algorithm

01:17:17.460 | on top of the Monte Carlo tree search.

01:17:19.100 | So we approximate this planning part by sampling.

01:17:23.220 | And it's successful on some small toy problems.

01:17:28.220 | We don't want to lose the generality, right?

01:17:33.520 | And that's sort of the handicap, right?

01:17:34.940 | If you want to be general, you have to give up something.

01:17:38.900 | So, but this single agent was able to play,

01:17:41.140 | you know, small games like coon poker and tic-tac-toe

01:17:44.260 | and even Pac-Man.

01:17:47.180 | And it's the same architecture, no change.

01:17:52.060 | The agent doesn't know the rules of the game,

01:17:54.940 | virtually nothing, all by itself,

01:17:56.780 | or by player with these environments.

01:17:58.860 | - So, Juergen Schmidhuber proposed something

01:18:03.220 | called Gate-On Machines,

01:18:04.780 | which is a self-improving program

01:18:06.940 | that rewrites its own code.

01:18:09.460 | Sort of mathematically or philosophically,

01:18:12.860 | what's the relationship in your eyes,

01:18:15.140 | if you're familiar with it,

01:18:16.220 | between Aixie and the Gate-On Machines?

01:18:18.460 | - Yeah, familiar with it.

01:18:19.780 | He developed it while I was in his lab.

01:18:22.380 | Yeah, so the Gate-On Machine, to explain it briefly,

01:18:26.260 | you give it a task.

01:18:28.980 | It could be a simple task as, you know,

01:18:30.460 | finding prime factors and numbers, right?

01:18:32.540 | You can formally write it down.

01:18:33.860 | There's a very slow algorithm to do that.

01:18:35.300 | Just try all the factors, yeah?

01:18:37.540 | Or play chess, right?

01:18:39.300 | Optimally, you write the algorithm to minimax

01:18:41.260 | to the end of the game,

01:18:42.140 | so you write down what the Girdle machine should do.

01:18:45.420 | Then it will take part of its resources

01:18:48.980 | to run this program,

01:18:50.820 | and other part of its sources to improve this program.

01:18:54.100 | And when it finds an improved version

01:18:56.980 | which provably computes the same answer,

01:19:00.780 | so that's the key part, yeah?

01:19:02.420 | It needs to prove by itself that this change of program

01:19:05.780 | still satisfies the original specification.

01:19:09.020 | And if it does so,

01:19:10.180 | then it replaces the original program

01:19:11.740 | by the improved program,

01:19:13.220 | and by definition does the same job, but just faster, okay?

01:19:17.140 | And then, you know, it proves over it and over it.

01:19:19.260 | And it's developed in a way that

01:19:22.380 | all parts of this Girdle machine can self-improve,

01:19:26.820 | but it stays provably consistent

01:19:29.220 | with the original specification.

01:19:31.860 | So from this perspective, it has nothing to do with IXE,

01:19:36.140 | but if you would now put IXE as the starting axioms in,

01:19:40.580 | it would run IXE, but, you know, that takes forever.

01:19:44.860 | But then if it finds a provable speedup of IXE,

01:19:48.540 | it would replace it by this, and this, and this,

01:19:51.020 | and maybe eventually it comes up with a model

01:19:52.900 | which is still the IXE model.

01:19:54.540 | It cannot be, I mean, just for the knowledgeable reader,

01:19:59.660 | IXE is incomputable, and I can prove that,

01:20:02.900 | therefore, there cannot be a computable exact

01:20:06.180 | algorithm computer.

01:20:08.660 | There needs to be some approximations,

01:20:10.380 | and this is not dealt with the Girdle machine,

01:20:12.020 | so you have to do something about it.

01:20:13.260 | But there's the IXETL model, which is finitely computable,

01:20:15.700 | which we could put in.

01:20:16.540 | - Which part of IXE is non-computable?

01:20:19.260 | - The Solomonov induction part.

01:20:20.780 | - The induction, okay, so.

01:20:22.260 | - But there is ways of getting computable approximations

01:20:26.380 | of the IXE model.

01:20:28.500 | So then it's at least computable.

01:20:30.060 | It is still way beyond any resources anybody will ever have,

01:20:33.740 | but then the Girdle machine could sort of improve it

01:20:35.900 | further and further in an exact way.

01:20:37.780 | - So is it theoretically possible

01:20:41.220 | that the Girdle machine process could improve?

01:20:45.020 | Isn't IXE already optimal?

01:20:51.900 | - It is optimal in terms of the revert collected

01:20:56.820 | over its interaction cycles,

01:20:59.420 | but it takes infinite time to produce one action.

01:21:03.540 | And the world continues whether you want it or not.

01:21:07.220 | So the model is, assuming you had an oracle

01:21:09.780 | which solved this problem,

01:21:11.260 | and then in the next 100 milliseconds

01:21:12.980 | or the reaction time you need gives the answer,

01:21:15.420 | then IXE is optimal.

01:21:16.700 | - Oh, so--

01:21:18.660 | - It's optimal in sense of data,

01:21:19.740 | also from learning efficiency and data efficiency,

01:21:23.740 | but not in terms of computation time.

01:21:25.660 | - And then the Girdle machine in theory,

01:21:27.580 | but probably not provably could make it go faster.

01:21:31.020 | - Yes.

01:21:31.860 | - Okay.

01:21:32.700 | Interesting.

01:21:34.660 | Those two components are super interesting.

01:21:36.660 | The sort of the perfect intelligence

01:21:39.360 | combined with self-improvement.

01:21:42.900 | Sort of provable self-improvement

01:21:45.540 | in the sense you're always getting the correct answer

01:21:48.780 | and you're improving.

01:21:50.340 | Beautiful ideas.

01:21:51.400 | Okay, so you've also mentioned

01:21:53.540 | that different kinds of things

01:21:55.740 | in the chase of solving this reward,

01:21:59.860 | sort of optimizing for the goal,

01:22:01.740 | interesting human things could emerge.

01:22:04.980 | So is there a place for consciousness within IXE?

01:22:08.820 | Where does, maybe you can comment,

01:22:13.500 | because I suppose we humans are just

01:22:16.380 | another instantiation of IXE agents

01:22:18.260 | and we seem to have consciousness.

01:22:20.900 | - You say humans are an instantiation of an IXE agent?

01:22:23.420 | - Yes.

01:22:24.260 | - Well, that would be amazing,

01:22:25.260 | but I think that's not really for the smartest

01:22:27.880 | and most rational humans.

01:22:29.000 | I think maybe we are very crude approximations.

01:22:32.920 | - Interesting.

01:22:33.760 | I mean, I tend to believe, again, I'm Russian,

01:22:35.740 | so I tend to believe our flaws are part of the optimal.

01:22:40.740 | So we tend to laugh off and criticize our flaws

01:22:45.660 | and I tend to think that that's actually close

01:22:49.260 | to an optimal behavior.

01:22:50.700 | But some flaws, if you think more carefully about it,

01:22:53.780 | are actually not flaws, yeah,

01:22:55.020 | but I think there are still enough flaws.

01:22:57.800 | - I don't know.

01:23:00.020 | It's unclear.

01:23:00.860 | As a student of history, I think all the suffering

01:23:03.200 | that we've endured as a civilization,

01:23:06.820 | it's possible that that's the optimal amount of suffering

01:23:10.240 | we need to endure to minimize long-term suffering.

01:23:13.840 | - That's your Russian background, I think.

01:23:17.300 | - That's the Russian,

01:23:18.140 | whether we humans are or not instantiations of an AIC agent,

01:23:21.900 | do you think consciousness is something that could emerge

01:23:25.700 | in a computational form of framework like AIC?

01:23:28.620 | - Let me also ask you a question.

01:23:31.740 | Do you think I'm conscious?

01:23:33.100 | - That's a good question.

01:23:38.060 | That tie is confusing me, but I think so.

01:23:44.300 | - You think that makes me unconscious

01:23:45.780 | because it strangles me?

01:23:47.180 | - If an agent were to solve the imitation game

01:23:49.780 | posed by Turing, I think they would be dressed

01:23:51.740 | similarly to you, because there's a kind of flamboyant,

01:23:56.740 | interesting, complex behavior pattern

01:24:01.060 | that sells that you're human and you're conscious.

01:24:04.500 | But why do you ask?

01:24:06.140 | - Was it a yes or was it a no?

01:24:07.940 | - Yes, I think you're--

01:24:08.780 | - Yes. (laughs)

01:24:10.300 | - I think you're conscious, yes.

01:24:12.220 | - Yeah, and you explain somehow why.

01:24:16.140 | But you infer that from my behavior, right?

01:24:18.660 | - Yes.

01:24:19.500 | - You can never be sure about that.

01:24:20.740 | And I think the same thing will happen

01:24:23.340 | with any intelligent agent we develop

01:24:26.780 | if it behaves in a way sufficiently close to humans,

01:24:31.060 | or maybe even not humans.

01:24:32.100 | I mean, maybe a dog is also sometimes

01:24:34.260 | a little bit self-conscious, right?

01:24:36.500 | So if it behaves in a way where we attribute

01:24:39.940 | typically consciousness, we would attribute consciousness

01:24:42.780 | to these intelligent systems and, you know,

01:24:45.340 | I see probably in particular.

01:24:47.300 | That, of course, doesn't answer the question

01:24:48.860 | whether it's really conscious.

01:24:50.860 | And that's the big, hard problem of consciousness.

01:24:53.540 | You know, maybe I'm a zombie.

01:24:55.700 | I mean, not the movie zombie, but the philosophical zombie.

01:24:59.360 | - Is, to you, the display of consciousness

01:25:02.620 | close enough to consciousness from a perspective of AGI

01:25:06.740 | that the distinction of the hard problem of consciousness

01:25:09.780 | is not an interesting one?

01:25:11.340 | - I think we don't have to worry

01:25:12.500 | about the consciousness problem,

01:25:13.980 | especially the hard problem for developing AGI.

01:25:16.860 | I think, you know, we progress.

01:25:20.220 | At some point we have, you know,

01:25:21.380 | solved all the technical problems

01:25:23.340 | and this system will behave intelligent

01:25:25.460 | and then super intelligent

01:25:26.540 | and this consciousness will emerge.

01:25:30.220 | I mean, definitely it will display behavior

01:25:32.540 | which we will interpret as conscious.

01:25:35.100 | And then it's a philosophical question.

01:25:38.180 | Did this consciousness really emerge

01:25:39.900 | or is it a zombie which just, you know, fakes everything?

01:25:43.740 | We still don't have to figure that out,

01:25:45.260 | although it may be interesting,

01:25:47.540 | at least from a philosophical point of view,

01:25:48.980 | it's very interesting,

01:25:49.900 | but it may also be sort of practically interesting.

01:25:53.220 | You know, there's some people saying,

01:25:54.300 | if it's just faking consciousness and feelings,

01:25:56.260 | you know, then we don't need to be concerned about rights.

01:25:59.180 | But if it's real conscious and has feelings,

01:26:01.660 | then we need to be concerned, yeah.

01:26:03.460 | - I can't wait till the day

01:26:07.580 | where AI systems exhibit consciousness

01:26:10.700 | because it'll truly be

01:26:12.460 | some of the hardest ethical questions

01:26:14.580 | of what we do with that.

01:26:15.700 | - It is rather easy to build systems

01:26:18.940 | which people ascribe consciousness.

01:26:21.140 | And I give you an analogy.

01:26:22.660 | I mean, remember, maybe it was before you were born,

01:26:25.380 | the Tamagotchi.

01:26:26.380 | - How dare you, sir?

01:26:29.860 | - Why, that's the,

01:26:31.980 | yeah, but you're young, right?

01:26:33.300 | - Yes, it's good to think, yeah, thank you.

01:26:34.980 | Thank you very much.

01:26:36.220 | But I was also in the Soviet Union.

01:26:37.580 | We didn't have any of those fun things.

01:26:41.260 | But you have heard about this Tamagotchi,

01:26:42.700 | which was really, really primitive,

01:26:44.620 | actually for the time it was,

01:26:46.980 | and you could raise this,

01:26:48.860 | and kids got so attached to it

01:26:51.660 | and didn't want to let it die.

01:26:53.620 | And I would have probably,

01:26:55.660 | if we would have asked the children,

01:26:57.460 | do you think this Tamagotchi is conscious?

01:26:59.540 | - They would have said yes.

01:27:00.380 | - Probably would have said yes, I would guess.

01:27:01.660 | - I think that's kind of a beautiful thing, actually,

01:27:04.780 | 'cause that consciousness, ascribing consciousness,

01:27:08.700 | seems to create a deeper connection.

01:27:10.500 | - Yep.

01:27:11.340 | - Which is a powerful thing,

01:27:12.660 | but we have to be careful on the ethics side of that.

01:27:15.940 | Well, let me ask about the AGI community broadly.

01:27:18.500 | You kind of represent some of the most serious work on AGI,

01:27:22.740 | at least earlier,

01:27:24.260 | and DeepMind represents serious work on AGI these days.

01:27:29.260 | But why, in your sense, is the AGI community so small,

01:27:34.100 | or has been so small, until maybe DeepMind came along?

01:27:38.100 | Like, why aren't more people seriously working

01:27:41.660 | on human-level and superhuman-level intelligence

01:27:45.860 | from a formal perspective?

01:27:47.380 | - Okay, from a formal perspective,

01:27:49.660 | that's sort of an extra point.

01:27:52.540 | So I think there are a couple of reasons.

01:27:54.940 | I mean, AI came in waves, right?

01:27:56.660 | You know, AI winters and AI summers,

01:27:58.500 | and then there were big promises which were not fulfilled.

01:28:01.500 | And people got disappointed,

01:28:05.740 | and that narrow AI, solving particular problems

01:28:10.740 | which seemed to require intelligence,

01:28:14.020 | was always, to some extent, successful,

01:28:17.020 | and there were improvements, small steps.

01:28:19.500 | And if you build something which is, you know,

01:28:22.060 | useful for society or industrially useful,

01:28:25.140 | then there's a lot of funding.

01:28:26.620 | So I guess it was in parts the money,

01:28:28.540 | which drives people to develop specific systems,

01:28:34.220 | solving specific tasks.

01:28:36.260 | But you would think that, you know,

01:28:37.380 | at least in university,

01:28:38.780 | you should be able to do ivory tower research.

01:28:42.860 | And that was probably better a long time ago,

01:28:46.020 | but even nowadays, there's quite some pressure

01:28:48.340 | of doing applied research or translational research,

01:28:52.260 | and, you know, it's harder to get grants as a theorist.

01:28:56.700 | So that also drives people away.

01:28:59.940 | It's maybe also harder,

01:29:01.540 | attacking the general intelligence problem.

01:29:03.180 | So I think enough people, I mean, maybe a small number,

01:29:05.940 | were still interested in formalizing intelligence

01:29:09.620 | and thinking of general intelligence,

01:29:12.940 | but, you know, not much came up, right?

01:29:17.620 | Or not much great stuff came up.

01:29:19.900 | - So what do you think,

01:29:21.380 | we talked about the formal big light

01:29:24.860 | at the end of the tunnel,

01:29:26.180 | but from the engineering perspective,

01:29:27.620 | what do you think it takes to build an AGI system?

01:29:30.380 | Is it, and I don't know if that's a stupid question

01:29:33.900 | or a distinct question

01:29:35.100 | from everything we've been talking about at IAXE,

01:29:37.100 | but what do you see as the steps

01:29:39.140 | that are necessary to take

01:29:41.020 | to start to try to build something?

01:29:43.020 | - So you want a blueprint now,

01:29:44.340 | and then you go off and do it?

01:29:46.340 | - That's the whole point of this conversation,

01:29:48.020 | I'm trying to squeeze that in there.

01:29:49.780 | Now, is there, I mean, what's your intuition?

01:29:51.580 | Is it in the robotics space

01:29:53.980 | or something that has a body

01:29:55.340 | and tries to explore the world?

01:29:56.820 | Is it in the reinforcement learning space,

01:29:58.940 | like the efforts with AlphaZero and AlphaStar

01:30:00.980 | that are kind of exploring

01:30:02.860 | how you can solve it through in the simulation,

01:30:05.500 | in the gaming world?

01:30:06.740 | Is there stuff in sort of

01:30:08.220 | all the transformer work in natural language processing,

01:30:13.220 | sort of maybe attacking the open domain dialogue?

01:30:15.780 | Like what, where do you see the promising pathways?

01:30:18.700 | - Let me pick the embodiment maybe.

01:30:24.540 | So, embodiment is important, yes and no.

01:30:29.540 | I don't believe that we need a physical robot

01:30:38.180 | walking or rolling around,

01:30:40.260 | interacting with the real world in order to achieve AGI.

01:30:45.100 | And I think it's more of a distraction

01:30:50.100 | probably than helpful.

01:30:51.380 | It's sort of confusing the body with the mind.

01:30:54.580 | For industrial applications or near-term applications,

01:30:58.940 | of course we need robots for all kinds of things,

01:31:01.180 | but for solving the big problem,

01:31:04.140 | at least at this stage, I think it's not necessary.

01:31:08.140 | But the answer is also yes,

01:31:10.100 | that I think the most promising approach

01:31:13.220 | is that you have an agent

01:31:15.300 | and that can be a virtual agent in a computer

01:31:18.500 | interacting with an environment,

01:31:20.140 | possibly a 3D simulated environment

01:31:22.580 | like in many computer games.

01:31:24.140 | And you train and learn the agent.

01:31:28.900 | Even if you don't intend to later put it sort of,

01:31:33.140 | this algorithm in a robot brain

01:31:35.580 | and leave it forever in the virtual reality,

01:31:38.580 | getting experience in a,

01:31:40.540 | although it's just simulated 3D world,

01:31:43.740 | is possibly, and I say possibly,

01:31:48.020 | important to understand things

01:31:51.700 | on a similar level as humans do,

01:31:53.980 | especially if the agent,

01:31:56.340 | or primarily if the agent needs to interact

01:31:59.540 | with the humans, right?

01:32:00.380 | You know, if you talk about objects

01:32:01.620 | on top of each other in space and flying in cars and so on,

01:32:04.820 | and the agent has no experience

01:32:06.500 | with even virtual 3D worlds,

01:32:09.660 | it's probably hard to grasp.

01:32:11.220 | So if you develop an abstract agent,

01:32:14.620 | say we take the mathematical path

01:32:16.820 | and we just want to build an agent

01:32:18.420 | which can prove theorems

01:32:19.580 | and becomes a better and better mathematician,

01:32:21.860 | then this agent needs to be able

01:32:23.860 | to reason in very abstract spaces

01:32:26.020 | and then maybe sort of putting it

01:32:27.780 | into 3D environment simulated

01:32:29.500 | or it is even harmful.

01:32:30.540 | It should sort of, you put it in,

01:32:32.740 | I don't know, an environment

01:32:34.060 | which it creates itself or so.

01:32:35.900 | - It seems like you have a interesting,

01:32:38.500 | rich, complex trajectory through life

01:32:40.740 | in terms of your journey of ideas.

01:32:42.740 | So it's interesting to ask what books,

01:32:45.860 | technical fiction, philosophical books,

01:32:49.580 | ideas, people had a transformative effect.

01:32:52.780 | Books are most interesting

01:32:53.900 | 'cause maybe people could also read those books

01:32:57.340 | and see if they could be inspired as well.

01:33:00.180 | - Yeah, luckily I asked books and not singular book.

01:33:03.580 | It's very hard and I tried to pin down one book.

01:33:08.180 | And I can do that at the end.

01:33:10.620 | So the most, the books which were most transformative

01:33:16.020 | for me or which I can most highly recommend

01:33:19.660 | to people interested in AI.

01:33:21.980 | - Both perhaps.

01:33:22.940 | - Yeah, yeah, both, yeah, yeah.

01:33:25.500 | I would always start with Russell and Norbeck,

01:33:28.620 | Artificial Intelligence, A Modern Approach.

01:33:30.940 | That's the AI Bible.

01:33:33.460 | It's an amazing book.

01:33:35.020 | It's very broad.

01:33:36.380 | It covers all approaches to AI.

01:33:38.900 | And even if you focus on one approach,

01:33:40.900 | I think that is the minimum you should know

01:33:42.580 | about the other approaches out there.

01:33:44.660 | So that should be your first book.

01:33:46.260 | - Fourth edition should be coming out soon.

01:33:48.380 | - Oh, okay, interesting.

01:33:50.100 | - There's a deep learning chapter now,

01:33:51.580 | so there must be.

01:33:53.140 | Written by Ian Goodfellow, okay.

01:33:55.620 | - And then the next book I would recommend,

01:33:59.740 | the Reinforcement Learning Book by Sutton and Bartow.

01:34:02.980 | That's a beautiful book.

01:34:04.500 | If there's any problem with the book,

01:34:07.940 | it makes RL feel and look much easier than it actually is.

01:34:13.820 | It's a very gentle book.

01:34:15.660 | It's very nice to read, the exercises to do.

01:34:17.620 | You can very quickly get some RL systems to run,

01:34:20.380 | you know, on very toy problems, but it's a lot of fun.

01:34:23.300 | And in a couple of days, you feel you know what RL is about,

01:34:28.300 | but it's much harder than the book.

01:34:31.460 | - Come on now, it's an awesome book.

01:34:34.860 | - Yeah, no, it is, yeah.

01:34:36.260 | And maybe, I mean, there's so many books out there.

01:34:41.260 | If you like the information theoretic approach,

01:34:43.460 | then there's "Colmogorov Complexity" by Leon Bitani,

01:34:46.780 | but probably, you know, some short article is enough.

01:34:50.820 | You don't need to read the whole book,

01:34:52.140 | but it's a great book.

01:34:54.420 | And if you have to mention one all-time favorite book,

01:34:59.420 | so different flavor, that's a book which is used

01:35:03.220 | in the International Baccalaureate

01:35:05.220 | for high school students in several countries.

01:35:09.020 | That's from Nicholas Alchen, "Theory of Knowledge."

01:35:12.940 | Second edition or first, not the third, please.

01:35:16.580 | The third one, they took out all the fun.

01:35:18.900 | So this asks all the interesting,

01:35:25.620 | or to me, interesting philosophical questions

01:35:27.620 | about how we acquire knowledge from all perspectives,

01:35:29.940 | you know, from math, from art, from physics,

01:35:32.100 | and ask how can we know anything?

01:35:36.220 | And the book is called "Theory of Knowledge."

01:35:38.020 | - From which, it's almost like a philosophical exploration

01:35:40.700 | of how we get knowledge from anything.

01:35:43.140 | - Yes, yeah, I mean, can religion tell us, you know,

01:35:45.140 | about something about the world?

01:35:46.180 | Can science tell us something about the world?

01:35:48.060 | Can mathematics, or is it just playing with symbols?

01:35:50.740 | And, you know, it's open-ended questions,

01:35:54.380 | and I mean, it's for high school students,

01:35:56.220 | so they have the resources from "Hitchhiker's Guide

01:35:58.300 | to the Galaxy" and from "Star Wars"

01:35:59.940 | and "The Chicken Crossed the Road," yeah?

01:36:01.780 | And it's fun to read, but it's also quite deep.

01:36:07.580 | - If you could live one day of your life over again,

01:36:11.460 | does it make you truly happy,

01:36:12.780 | or maybe like we said with the books,

01:36:14.380 | it was truly transformative.

01:36:16.180 | What day, what moment would you choose?

01:36:19.060 | Does something pop into your mind?

01:36:20.780 | - Does it need to be a day in the past,

01:36:23.420 | or can it be a day in the future?

01:36:25.860 | - Well, space-time is an emergent phenomena,

01:36:27.900 | so it's all the same anyway.

01:36:30.340 | - Okay.

01:36:31.180 | Okay, from the past.

01:36:33.340 | - You're really gonna say from the future, I love it.

01:36:36.740 | No, I will tell you from the future, yeah?

01:36:38.300 | - Okay, from the past.

01:36:39.140 | - So from the past, I would say

01:36:41.380 | when I discovered my axiom model.

01:36:43.700 | I mean, it was not in one day,

01:36:45.100 | but it was one moment where I realized

01:36:48.780 | Kolmogorov complexity,

01:36:49.900 | I didn't even know that it existed,

01:36:53.100 | but I discovered sort of this compression idea myself,

01:36:56.220 | but immediately I knew I can't be the first one,

01:36:58.020 | but I had this idea,

01:36:59.220 | and then I knew about sequential decision tree,

01:37:02.100 | and I knew if I put it together, this is the right thing.

01:37:06.260 | And yeah, still when I think back about this moment,

01:37:09.580 | I'm super excited about it.

01:37:12.300 | - Was there any more details and context that moment?

01:37:16.220 | Did an apple fall on your head?

01:37:17.900 | So like if you look at Ian Goodfellow talking about GANs,

01:37:23.980 | there was beer involved.

01:37:25.820 | Is there some more context of what sparked your thought,

01:37:30.060 | or was it just--

01:37:31.060 | - No, it was much more mundane.

01:37:32.820 | So I worked in this company,

01:37:34.460 | so in this sense, the four and a half years

01:37:36.020 | was not completely wasted.

01:37:37.500 | And I worked on an image interpolation problem,

01:37:43.620 | and I developed a quite neat new interpolation techniques,

01:37:48.340 | and they got patented,

01:37:49.380 | and then which happens quite often,

01:37:52.100 | I got sort of overboard and thought about,

01:37:54.260 | yeah, that's pretty good, but it's not the best,

01:37:56.100 | so what is the best possible way of doing interpolation?

01:37:59.660 | And then I thought, yeah, you want the simplest picture,

01:38:03.060 | which if you core screen it,

01:38:04.620 | recovers your original picture,

01:38:06.420 | and then I thought about the simplicity concept

01:38:08.740 | more in quantitative terms,

01:38:11.140 | and yeah, then everything developed.

01:38:13.940 | - And somehow the full beautiful mix

01:38:16.940 | of also being a physicist

01:38:18.780 | and thinking about the big picture of it

01:38:20.500 | then led you to probably--

01:38:22.300 | - Yeah, yeah, so as a physicist,

01:38:25.100 | I was probably trained not to always think

01:38:26.900 | in computational terms, just ignore that

01:38:29.220 | and think about the fundamental properties

01:38:32.220 | which you want to have.

01:38:34.060 | - So what about if you could really one day in the future,

01:38:36.980 | what would that be?

01:38:39.940 | - When I solve the AGI problem?

01:38:41.540 | - I don't think-- - In practice, in practice,

01:38:44.580 | so in theory I have solved it with the Ix-A model,

01:38:46.460 | but in practice. - Yes.

01:38:47.700 | - And then I ask the first question.

01:38:50.780 | - What would be the first question?

01:38:53.300 | - What's the meaning of life?

01:38:54.740 | - I don't think there's a better way to end it.

01:38:58.500 | Thank you so much for talking today,

01:38:59.340 | it's a huge honor to finally meet you.

01:39:01.420 | - Yeah, thank you too, it was a pleasure of mine, too.

01:39:04.620 | - Thanks for listening to this conversation

01:39:06.300 | with Marcus Hutter,

01:39:07.420 | and thank you to our presenting sponsor, Cash App.

01:39:10.060 | Download it, use code LEXPODCAST,

01:39:12.620 | you'll get $10, and $10 will go to FIRST,

01:39:15.500 | an organization that inspires and educates young minds

01:39:18.560 | to become science and technology innovators of tomorrow.

01:39:22.340 | If you enjoy this podcast, subscribe on YouTube,

01:39:25.140 | get five stars on Apple Podcasts,

01:39:27.020 | support on Patreon, or simply connect with me on Twitter

01:39:30.420 | at Lex Friedman.

01:39:32.040 | And now, let me leave you with some words of wisdom

01:39:35.740 | from Albert Einstein.

01:39:37.900 | "The measure of intelligence is the ability to change."

01:39:42.120 | Thank you for listening, and hope to see you next time.

01:39:46.500 | (upbeat music)

01:39:49.080 | (upbeat music)

01:39:51.660 | [BLANK_AUDIO]

Marcus Hutter: Universal Artificial Intelligence, AIXI, and AGI | Lex Fridman Podcast #75

Chapters