Deep Learning State of the Art (2020)

00:00:00.000 | Welcome to 2020 and welcome to the Deep Learning Lecture Series.

00:00:07.500 | Let's start it off today to take a quick whirlwind tour of all the exciting things that happened

00:00:14.060 | in '17, '18, and '19 especially, and the amazing things we're going to see this year in 2020.

00:00:23.420 | Also as part of this series, there's going to be a few talks from some of the top people

00:00:28.760 | in learning in artificial intelligence after today, of course.

00:00:34.000 | Start at the broad, the celebrations from the Turing Award to the limitations and the

00:00:39.000 | debates and the exciting growth first.

00:00:44.300 | And first, of course, a step back to the quote I've used before.

00:00:47.040 | I love it.

00:00:48.040 | I'll keep reusing it.

00:00:49.320 | "AI began not with Alan Turing or McCarthy, but with the ancient wish to forge the gods,"

00:00:57.160 | a quote from Pamela McCordick in Machines Who Think.

00:01:01.160 | That visualization there is just 3% of the neurons in our brain of the thalamocortical

00:01:08.040 | system.

00:01:10.640 | That magical thing between our ears that allows us all to see and hear and think and reason

00:01:16.040 | and hope and dream and fear our eventual mortality, all of that is the thing we wish to understand.

00:01:25.300 | That's the dream of artificial intelligence and recreate versions of it, echoes of it

00:01:32.840 | in engineering of our intelligence systems.

00:01:36.660 | That's the dream.

00:01:37.660 | We should never forget in the details I'll talk, the exciting stuff I'll talk about today.

00:01:41.600 | That's sort of the reason why this is exciting, this mystery that's our mind.

00:01:49.660 | The modern human brain, the modern human as we know them today, know and love them today,

00:01:56.080 | it's just about 300,000 years ago.

00:01:59.460 | And the Industrial Revolution is about 300 years ago.

00:02:02.620 | That's 0.1% of the development since the early modern human being is when we've seen a lot

00:02:11.880 | of the machinery.

00:02:13.620 | The machine was born not in stories, but in actuality.

00:02:19.340 | The machine was engineered since the Industrial Revolution and the steam engine and the mechanized

00:02:24.140 | factory system and the machining tools.

00:02:26.500 | That's just 0.1% in the history.

00:02:28.940 | That's the 300 years.

00:02:30.480 | Now we zoom in to the 60, 70 years since the founder, the father arguably of artificial

00:02:36.980 | intelligence, Alan Turing and the dreams.

00:02:40.540 | There's always been the dance in artificial intelligence between the dreams, the mathematical

00:02:44.580 | foundations and when the dreams meet the engineering, the practice, the reality.

00:02:51.660 | So Alan Turing has spoken many times that by the year 2000 that he would be sure that

00:02:58.220 | the Turing test, natural language would be passed.

00:03:01.300 | It seems probably, he said, that once the machine thinking method has started, it would

00:03:05.860 | not take long to outstrip our feeble powers.

00:03:09.680 | They would be able to converse with each other, to sharpen their wits.

00:03:14.080 | Some stage therefore, we should have to expect the machines to take control.

00:03:18.240 | A little shout out to self-play there.

00:03:22.880 | So that's the dream, both the father of the mathematical foundation of artificial intelligence

00:03:27.320 | and the father of dreams in artificial intelligence.

00:03:30.840 | And that dream, again, in the early days was taking reality, the practice met with the

00:03:35.480 | perceptron, often thought of as a single layer neural network, but actually what's not as

00:03:43.160 | much known as Frank Rosenblatt was also the developer of the multi-layer perceptron.

00:03:49.080 | And that history zooming through has amazed our civilization.

00:03:53.280 | To me, one of the most inspiring things in the world of games, first with the great Garry

00:03:58.720 | Kasparov losing to IBM Dblue in 1997, then Lee Sedol losing to AlphaGo in 2016, seminal

00:04:08.320 | moments and captivating the world through the engineering of actual real world systems.

00:04:14.920 | Robots on four wheels, as we'll talk about today, from Waymo to Tesla to all of the autonomous

00:04:20.320 | vehicle companies working in the space, robots on two legs, captivating the world of what

00:04:26.920 | actuation, what kind of manipulation can be achieved.

00:04:32.400 | The history of deep learning from 1943, the initial models from neuroscience, thinking

00:04:40.760 | about neural networks, how to model neural networks mathematically to the creation, as

00:04:45.600 | I said, of the single layer and the multi-layer perceptron by Frank Rosenblatt in '57 and

00:04:50.800 | '62, to the ideas of back propagation and recurring neural nets in the '70s and '80s,

00:04:56.920 | to convolutional neural networks and LSTMs and bidirectional RNNs in the '80s and '90s,

00:05:02.680 | to the birth of the deep learning term and the new wave, the revolution in 2006, to the

00:05:11.240 | ImageNet and AlexNet, the seminal moment that captivated the possibility, the imagination

00:05:16.600 | of the AI community of what neural networks can do in the image and natural language space

00:05:23.320 | closely following years after, to the development, to the popularization of GANs, generative

00:05:30.480 | adversarial networks, with AlphaGo and AlphaZero in 2016 and '7.

00:05:35.360 | And as we'll talk about, language models of transformers in '17, '18, and '19, those

00:05:41.440 | have been the last few years, have been dominated by the ideas of deep learning in the space

00:05:45.900 | of natural language processing.

00:05:47.960 | OK, celebrations.

00:05:50.000 | This year, the Turing Award was given for deep learning.

00:05:53.400 | This is like deep learning has grown up.

00:05:55.640 | We can finally start giving awards.

00:05:57.840 | Jan LeCun, Jeffrey Hinton, Yosha Bengel received the Turing Award for their conceptual engineering

00:06:04.560 | breakthroughs that have made deep neural networks a critical component of computing.

00:06:10.000 | I would also like to add that perhaps the popularization in the face of skepticism,

00:06:15.120 | and for those a little bit older have known the skepticism that neural networks have received

00:06:19.240 | throughout the '90s, in the face of that skepticism, continuing pushing, believing, and working

00:06:24.520 | in this field, and popularizing it through in the face of that skepticism, I think is

00:06:30.520 | part of the reason these three folks have received the award.

00:06:34.960 | But of course, the community that contributed to deep learning is bigger, much bigger than

00:06:39.400 | those three, many of whom might be here today at MIT, broadly in academia and industry.

00:06:48.520 | Looking at the early key figures, Walter Pitts and Warren McCulloch, as I mentioned, for

00:06:54.120 | the computational models of the neural nets, these ideas of thinking that the kind of neural

00:07:00.000 | networks, biological neural networks, can have on our brain can be modeled mathematically.

00:07:05.200 | And then the engineering of those models into actual physical and conceptual mathematical

00:07:11.680 | systems by Frank Rosenblatt, '57, again, single layer, multi-layer in 1962.

00:07:18.720 | You could say Frank Rosenblatt is the father of deep learning, the first person to really,

00:07:23.840 | in '62, mention the idea of multiple hidden layers in neural networks.

00:07:28.040 | As far as I know, somebody was correct me.

00:07:30.840 | In 1965, shout out to the Soviet Union and Ukraine, the person who is considered to be

00:07:39.360 | the father of deep learning, Alexei Evaknenko and V.G.

00:07:44.600 | Lapa, co-author of that work, is the first learning algorithms on multi-layer perceptrons,

00:07:51.240 | multiple hidden layers.

00:07:53.720 | The work on back propagation, automatic differentiation, 1970.

00:07:57.880 | In 1979, convolutional neural networks were first introduced.

00:08:02.480 | And John Hopfield, looking at recurrent neural networks, what are now called Hopfield networks,

00:08:06.960 | a special kind of recurrent neural networks.

00:08:08.920 | Okay, that's the early birth of deep learning.

00:08:12.600 | I want to mention this because it's been a kind of contention space, now that we can

00:08:17.160 | celebrate the incredible accomplishments of deep learning.

00:08:20.760 | Much like in reinforcement learning, in academia, credit assignment is a big problem.

00:08:26.440 | And the embodiment of that, almost a point of meme, is the great Juergen Schmidhuber.

00:08:34.520 | I encourage for people who are interested in the amazing contribution of the different

00:08:38.320 | people in the deep learning field to read his work on deep learning and neural networks.

00:08:42.360 | It's an overview of all the various people who have contributed besides Jan Lekun, Jeffrey

00:08:48.200 | Hinton, and Yoshua Bengio.

00:08:51.520 | It's a big, beautiful community.

00:08:55.040 | But full of great ideas and full of great people.

00:08:59.440 | My hope for this community, given the tension, as some of you might have seen, around this

00:09:04.480 | kind of credit assignment problem, is that we have more, not on this slide, but love.

00:09:13.720 | There can never be enough love in the world.

00:09:16.320 | But general respect, open-mindedness, and collaboration, and credit sharing in the community.

00:09:21.200 | Plus derision, jealousy, and stubbornness, and silos, academic silos, within institutions,

00:09:27.120 | within disciplines.

00:09:29.720 | Also 2019 was the first time it became cool to highlight the limits of deep learning.

00:09:38.220 | This is the interesting moment in time.

00:09:43.320 | Several books, several papers have come out in the past couple of years highlighting that

00:09:48.760 | deep learning is not able to do the kind of, the broad spectrum of tasks that we can think

00:09:53.920 | of the artificial intelligence system being able to do.

00:09:56.520 | Like read common sense reasoning, like building knowledge bases, and so on.

00:10:03.160 | Rodney Brooks said by 2020, the popular press starts having stories that the era of deep

00:10:08.440 | learning is over.

00:10:10.960 | And certainly there has been echoes of that through the press, through the Twitter sphere,

00:10:17.920 | and all that kind of world.

00:10:19.560 | And I'd like to say that a little skepticism, a little criticism is really good always for

00:10:25.680 | the community, but not too much.

00:10:27.200 | It's like a little spice in the soup of progress.

00:10:32.200 | Aside from that kind of skepticism, the growth of CVPR, iClear, NeurIPS, all these conference

00:10:42.880 | submission papers has grown year over year.

00:10:46.320 | There's been a lot of exciting research, some of which I'd like to cover today.

00:10:52.320 | My hope in this space of deep learning growth celebrations and limitations for 2020 is that

00:10:58.160 | there's less, both less hype and less anti-hype.

00:11:06.120 | Less tweets on how there's too much hype in AI and more solid research.

00:11:11.080 | Less criticism and more doing.

00:11:13.360 | But again, a little criticism, a little spice is always good for the recipe.

00:11:19.720 | Hybrid research, less contentious counterproductive debates and more open-minded interdisciplinary

00:11:26.040 | collaboration across neuroscience, cognitive science, computer science, robotics, mathematics,

00:11:34.440 | physics, across all these disciplines working together.

00:11:38.580 | And the research topics that I would love to see more contributions to, as we'll briefly

00:11:42.520 | talk about in some domains, is reasoning, common sense reasoning, integrating that into

00:11:47.160 | the learning architecture, active learning and lifelong learning, multimodal multitask

00:11:53.840 | learning, open domain conversation, so expanding the success of natural language to dialogue,

00:11:59.680 | to open domain dialogue and conversation, and then applications.

00:12:04.200 | The two most exciting, one of which we'll talk about is medical and autonomous vehicles.

00:12:09.800 | Then algorithmic ethics in all of its forms, fairness, privacy, bias.

00:12:15.080 | There's been a lot of exciting research there.

00:12:16.720 | I hope that continues.

00:12:18.320 | Taking responsibility for the flaws in our data and the flaws in our human ethics.

00:12:25.240 | And then robotics in terms of deep learning application robotics.

00:12:29.000 | I'd love to see a lot of development, continued development, deep reinforcement learning application

00:12:33.240 | in robotics and robot manipulation.

00:12:36.600 | By the way, there might be a little bit of time for questions at the end.

00:12:40.480 | If you have a really pressing question, you can ask it along the way too.

00:12:44.680 | Questions so far?

00:12:46.000 | Thank God.

00:12:48.720 | Okay.

00:12:49.960 | So, first, the practical.

00:12:53.360 | The deep learning and deep RL frameworks.

00:12:56.720 | This has really been a year where the frameworks have really matured and converged towards

00:13:02.320 | two popular deep learning frameworks that people have used.

00:13:06.640 | As TensorFlow and PyTorch.

00:13:08.120 | So TensorFlow 2.0 and PyTorch 1.3 is the most recent version.

00:13:14.280 | And they've converged towards each other, taking the best features, removing the weaknesses

00:13:18.400 | from each other.

00:13:19.920 | So that competition has been really fruitful in some sense for the development of the community.

00:13:25.960 | So on the TensorFlow side, eager execution.

00:13:28.360 | So imperative programming.

00:13:30.520 | How you would program in Python has become the default.

00:13:33.200 | Has been first integrated, made easy to use, and become the default.

00:13:38.800 | And on the PyTorch side, TorchScript allowed for now graph representation.

00:13:44.260 | So do what you used to be able to do and what used to be the default mode of operation in

00:13:48.960 | TensorFlow.

00:13:49.960 | Allow you to have this intermediate representation that's in graph form.

00:13:55.360 | On the TensorFlow side, just the deep Keras integration and promotion as the primary citizen,

00:14:04.720 | the default citizen of the API of the way you would track with TensorFlow.

00:14:10.760 | Allowing complete beginners, just anybody outside of machine learning to use TensorFlow

00:14:14.800 | with just a few lines of code to train and do inference with a model.

00:14:19.480 | That's really exciting.

00:14:20.480 | They cleaned up the API, the documentation, and so on.

00:14:23.520 | And of course, maturing the JavaScript in the browser implementation of TensorFlow,

00:14:29.320 | TensorFlow Lite, being able to run TensorFlow on phones, mobile, and serving.

00:14:34.160 | Apparently, this is something industry cares a lot about, of course, is being able to efficiently

00:14:40.600 | use models in the cloud.

00:14:44.520 | And PyTorch catching up with TPU support and experimental versions of PyTorch mobile.

00:14:49.640 | So being able to run a smartphone on their side.

00:14:51.960 | This tense, exciting competition.

00:14:53.680 | Oh, and I almost forgot to mention, we have to say goodbye to our favorite Python 2.

00:14:58.920 | This is the year that support finally, in the January 1st, 2020, support for Python

00:15:03.640 | 2 and TensorFlow's and PyTorch's support for Python 2 has ended.

00:15:08.960 | So goodbye, print.

00:15:11.240 | Goodbye, cruel world.

00:15:15.000 | On the reinforcement learning front, we're kind of in the same space as JavaScript libraries

00:15:19.680 | are in.

00:15:21.040 | There's no clear winners coming out.

00:15:23.040 | If you're a beginner in the space, the one I recommend is a fork of open app baselines

00:15:29.080 | is stable baselines.

00:15:31.800 | But there's a lot of exciting ones.

00:15:33.000 | Some of them are really closely built on TensorFlow.

00:15:35.480 | Some are built on PyTorch.

00:15:38.960 | Of course, from Google, from Facebook, from DeepMind, Dopamine, TF agents, TensorFlow.

00:15:47.360 | Most of these I've used if you have specific questions, I can answer them.

00:15:51.640 | So stable baselines is the open a baselines for because I said this implements a lot of

00:15:55.760 | the basic deep RL algorithms, PPO, etc, everything good documentation and just allows very simple,

00:16:03.640 | minimal few lines of code implementation of the basic, the matching of the basic algorithms

00:16:08.720 | of the open AI gym environments.

00:16:12.000 | That's the one I recommend.

00:16:13.320 | Okay, for the framework world, my hope for 2020 is framework agnostic research.

00:16:18.840 | So one of the things that I mentioned is PyTorch has really become almost overtaking TensorFlow

00:16:25.800 | in popularity in the research world.

00:16:29.080 | What I'd love to see is being able to develop an architecture in TensorFlow or developing

00:16:33.880 | in PyTorch, which you currently can.

00:16:35.560 | And then once you train the model to be able to easily transfer to the other from PyTorch

00:16:41.360 | TensorFlow and from TensorFlow to PyTorch, currently it takes three, four, five hours

00:16:45.480 | if you know what you're doing in both languages to do that.

00:16:47.960 | It'd be nice if there was a very easy way to do that transfer.

00:16:52.980 | Then the maturing of the deep RL frameworks, I'd love it to see open AI step up, DeepMind

00:16:57.960 | to step up and really take some of these frameworks to maturity that we can all agree on.

00:17:03.680 | Much like open AI gym for the environment world has done.

00:17:07.080 | And continued work that Keras has started and many other wrappers around TensorFlow

00:17:11.660 | has started of greater and greater abstractions, allowing machine learning to be used by people

00:17:17.000 | outside of the machine learning field.

00:17:19.540 | I think the powerful thing about supervised sort of basic vanilla supervised learning

00:17:26.940 | is that people in biology and chemistry and neuroscience and in physics, in astronomy

00:17:37.460 | can deal with a huge amount of data that they're working with and without needing to learn

00:17:43.420 | any of the details of even Python.

00:17:46.340 | So that I would love to see greater and greater abstractions, which empower scientists outside

00:17:51.820 | the field.

00:17:52.820 | Okay.

00:17:53.820 | Natural language processing, 2017, 2018 was in the transformer was developed and its power

00:18:05.820 | was demonstrated most, especially by Bert, achieving a lot of state of the art results

00:18:15.980 | on a lot of language benchmarks from sentence classification to tagging question answering

00:18:23.380 | and so on.

00:18:26.820 | There's hundreds of data sets and benchmarks that emerged.

00:18:30.560 | Most of which Bert has dominated in 2018, 2019 was sort of the year that the transformer

00:18:40.380 | really exploded in terms of all the different variations.

00:18:44.380 | Again, starting from Bert, Excel net is very cool to use Bert in the name of your new derivative

00:18:53.020 | of a transformer.

00:18:55.060 | Roberta distilled Bert from hugging face Salesforce, open AI, GPT two, of course, Albert and Megatron

00:19:03.580 | from Nvidia, huge transformer.

00:19:06.220 | A few tools have emerged.

00:19:07.860 | So one on hugging face is a company and also a repository that has implemented in both

00:19:14.340 | Pytorch and TensorFlow, a lot of these transformer based natural language models.

00:19:19.460 | So that's really exciting.

00:19:21.040 | So most people here can just use it easily.

00:19:24.140 | So those are already pre-trained models.

00:19:26.260 | And the other exciting stuff is Sebastian Ruder, great researcher in the field of natural

00:19:31.540 | language processing has put together NLP progress, which is all the different benchmarks for

00:19:36.940 | all the different natural language tasks, tracking who sort of leaderboards of who's

00:19:41.060 | winning where.

00:19:42.060 | Okay.

00:19:43.060 | I'll mention a few models that stand out the work from this year.

00:19:47.660 | Megatron LM from Nvidia is basically taking, I believe the GPT two transformer model and

00:19:54.020 | just putting it on steroids, right?

00:19:57.700 | 8.3 versus 1.5 billion parameters.

00:20:03.440 | And a lot of interesting stuff there, as you would expect from Nvidia, of course, there's

00:20:07.860 | always brilliant research, but also interesting aspects about how to train in a parallel way,

00:20:14.020 | model and data parallelism in the training.

00:20:17.260 | The first breakthrough results in terms of performance, the model that replaced BERT

00:20:23.680 | as king of transformers is XLNet from CMU and Google research.

00:20:29.940 | They combine the bidirectionality from BERT and the recurrence aspect of transformer XL,

00:20:38.300 | the relative positional embeddings and the recurrence mechanism of transformer XL to

00:20:43.900 | taking the bidirectionality and the recurrence combining it to achieve state of the art performance

00:20:48.860 | on 20 tasks.

00:20:50.620 | Albert is a recent addition from Google research and it reduces significantly the amount of

00:20:58.660 | parameters versus BERT by doing a parameter sharing across the layers.

00:21:03.220 | And it has achieved state of the art results on 12 NLP tasks, including the difficult Stanford

00:21:11.140 | question answering benchmark of squad two.

00:21:15.140 | And they provide open source TensorFlow implementation, including a number of ready to use pre-trained

00:21:20.420 | language models.

00:21:21.620 | Okay.

00:21:22.620 | Another way for people who are completely new to this field, a bunch of apps, right?

00:21:27.300 | With transformer is one of them from Hugging Face popped up that allows you to explore

00:21:33.300 | the capabilities of these language models.

00:21:35.220 | And I think they're quite fascinating from a philosophical point of view.

00:21:39.940 | And this, this has actually been at the core of a lot of the tension of how much do these

00:21:44.420 | transformers actually understand basically memorizing the statistics of the language

00:21:50.420 | in a self-supervised way by reading a lot of texts.

00:21:53.940 | Is that really understanding?

00:21:55.900 | A lot of people say no until it impresses us.

00:21:59.500 | And then everybody will say it's obvious, but right with transformer is a really powerful

00:22:04.500 | way to generate texts, to reveal to you how much these models really learn.

00:22:09.900 | Before this yesterday, actually just came up with a bunch of prompts.

00:22:14.540 | So on the left is a prompt you give it the meaning of life here, for example, is not

00:22:19.460 | what I think it is.

00:22:21.440 | It's what I do to make it.

00:22:22.780 | And you can do a lot of prompts of this nature is very profound.

00:22:26.620 | And some of them will be just absurd.

00:22:29.060 | You'll make sense of it statistically, but it'll be absurd in reveal that the model really

00:22:34.060 | doesn't understand the fundamentals of the prompt is being provided.

00:22:39.300 | But at the same time, it's incredible what kind of texts is able to generate.

00:22:47.380 | The limits of deep learning, I'm just having fun with this at this point, still are still

00:22:52.620 | in the process of being figured out.

00:22:54.420 | Very true.

00:22:55.900 | Had to type this most important person in the history of deep learning is probably Andrew

00:23:02.180 | Ang.

00:23:03.180 | I have to agree.

00:23:04.480 | So this model knows what it's doing.

00:23:07.900 | And I tried to get it to say something nice about me.

00:23:11.980 | And that's a lot of attempts.

00:23:14.100 | So this is kind of funny is finally it did one.

00:23:17.700 | I said, Lex Freeman's best qualities that he's smart.

00:23:23.340 | I said, finally, but it's never nothing but ever happens.

00:23:28.500 | But I think he gets more attention than every Twitter comment ever.

00:23:35.100 | That's very true.

00:23:36.540 | Okay, a nice way to sort of reveal through this, that the models are not able to do any

00:23:43.160 | kind of understanding of language is just to do prompts that show understanding of concepts

00:23:50.340 | and being able to reason with those concepts, common sense reasoning.

00:23:53.740 | Trivia one is doing two plus two is a three, five is a six, seven, the result is a simple

00:24:00.620 | equation four, and two plus three is like you got it right and then it changes mind.

00:24:05.860 | Okay, two minus two is seven.

00:24:08.580 | So on.

00:24:09.580 | You can reveal any kind of reasoning you can do a blocks, you can ask it about gravity,

00:24:13.980 | all those kinds of things.

00:24:14.980 | It shows that it doesn't understand the fundamentals of the concepts that are being reasoned about.

00:24:21.060 | And I'll mention of work that takes it beyond towards that reasoning world in the next few

00:24:28.260 | slides.

00:24:29.260 | But I should also mention with his GPT to model, if you remember about a year ago, there

00:24:33.820 | was a lot of thinking about this 1.5 billion parameter model from open AI.

00:24:39.660 | It is so the thought was it might be so powerful that it would be dangerous.

00:24:46.500 | And so the idea from open AI is when you have an AI system that you're about to release

00:24:51.780 | that might turn out to be dangerous, in this case used probably by Russians, fake news

00:24:59.660 | or misinformation, that that's the kind of thinking is how do we release it.

00:25:04.420 | And I think while it turned out that the GPT to model is not quite so dangerous, that humans

00:25:10.020 | are in fact more dangerous than AI currently.

00:25:13.740 | The that thought experiment is very interesting.

00:25:16.700 | They released a report on release strategies and the social impacts of language models

00:25:21.340 | that almost didn't get as much attention as I think it should.

00:25:25.900 | And it was a little bit disappointing to me how little people are worried about this kind

00:25:31.020 | of situation.

00:25:34.140 | It was more of an eye roll about, oh, these language models aren't as smart as as we thought

00:25:42.020 | they might be.

00:25:43.220 | But the reality is once they are, it's a very interesting thought experiment of how should

00:25:49.580 | the process go of companies and experts communicating with each other during that release.

00:25:54.680 | This report thinks through some of those details.

00:25:58.620 | My takeaway from just reading the report from this whole year of that event is that conversation

00:26:04.620 | on this topic are difficult because we as the public seem to penalize anybody trying

00:26:10.160 | to have that conversation.

00:26:12.340 | And the model of sharing privately, confidentially between ML, machine learning organizations

00:26:17.700 | and experts is not there.

00:26:19.980 | There's no incentive or model or history or culture of sharing.

00:26:25.260 | Okay.

00:26:26.900 | Best paper from ACL, the main conference for languages was on the difficult task of -- we

00:26:35.660 | talked about language models.

00:26:37.500 | Now there's the task taking it a step further of dialogue.

00:26:43.260 | Multidomain task-oriented dialogue.

00:26:46.060 | That's sort of like the next challenge for dialogue systems.

00:26:49.860 | And they've had a few ideas on how to perform dialogue state tracking across domains, achieving

00:26:57.740 | state of the art performance on multi-WAS, which is a five domain challenging, very difficult

00:27:04.380 | five domain human to human dialogue data set.

00:27:07.380 | There's a few ideas there.

00:27:08.860 | Should probably hurry up and start skipping stuff.

00:27:13.700 | And the common sense reasoning, which is really interesting, is this one of the open questions

00:27:19.380 | for the deep learning community, AI community in general, is how can we have hybrid systems

00:27:24.460 | of whether it's symbolic AI deep learning or generally common sense reasoning with learning

00:27:28.900 | systems.

00:27:29.900 | And there's been a few papers in this space.

00:27:32.020 | One of my favorites from Salesforce on building a data set where we can start to do question

00:27:40.420 | answering and figuring out the concepts that are being explored in the question and answering.

00:27:47.460 | Here the question while eating a hamburger with friends, what are people trying to do?

00:27:52.220 | Multiple choice, have fun, tasty, indigestion.

00:27:57.060 | The idea that needs to be generated there, and that's where the language model would

00:28:02.780 | come in, is that usually a hamburger with friends indicates a good time.

00:28:08.380 | So you basically take the question, generate the common sense concept, and from that be

00:28:16.300 | able to determine the multiple choice, what's happening, what's the state of affairs in

00:28:24.060 | this particular question.

00:28:28.420 | Alexa Prize, again, hasn't received nearly enough attention that I think it should have,

00:28:33.860 | perhaps because there hasn't been major breakthroughs, but it's open domain conversations that all

00:28:39.300 | of us, anybody who owns an Alexa can participate in as a provider of data.

00:28:49.460 | But there's been a lot of amazing work from universities across the world on the Alexa

00:28:54.140 | Prize in the last couple of years, and there's been a lot of interesting lessons summarized

00:28:57.940 | in papers and blog posts.

00:29:00.460 | A few lessons from Alquist's team that I particularly like, and this is kind of echoes the work

00:29:06.340 | in IBM Watson with the Jeopardy challenge, is that one of the big ones is that machine

00:29:13.580 | learning is not an essential tool for effective conversation yet.

00:29:19.100 | So machine learning is useful for general chit-chat when you fail at deep, meaningful

00:29:24.420 | conversation or actually understanding what the topic we're talking about, so throwing

00:29:28.460 | in chit-chat, and classification, sort of classifying intent, finding the entities,

00:29:34.620 | predicting the sentiment of the sentences, that's sort of an assistive tool.

00:29:39.220 | But the fundamentals of the conversation are the following.

00:29:43.140 | So first you have to break it apart.

00:29:45.700 | So conversation is a, you can think of it as a long dance, and the way you have fun

00:29:53.540 | dancing is you break it up into a set of moves and turns and so on, and focus on that, sort

00:29:59.020 | of live in the moment kind of thing.

00:30:00.860 | So focus on small parts of the conversation taken at a time.

00:30:04.460 | Then also have a graph, sort of conversation is also all about tangents, so have a graph

00:30:09.660 | of topics and be ready to jump context, from one context to the other and back.

00:30:15.740 | If you look at some of these natural language conversations that they publish, it's just

00:30:19.660 | all over the place in terms of topics.

00:30:22.020 | You jump back and forth, and that's the beauty, the humor, the wit, the fun of conversation

00:30:26.700 | is you jump around from topic to topic.

00:30:29.940 | And opinions, one of the things that natural language systems don't seem to have much is

00:30:34.300 | intelligence.

00:30:35.540 | If I learned anything, one of the simplest ways to convey intelligence is to be very

00:30:42.020 | opinionated about something and confident, and that's a really interesting concept about

00:30:48.100 | conversation.

00:30:49.100 | In general, there's just a lot of lessons.

00:30:50.780 | Oh, and finally, of course, maximize entertainment, not information.

00:30:55.260 | This is true for autonomous vehicles.

00:30:57.100 | This is true for natural language conversation is fun should be part of the objective function.

00:31:03.980 | That's the lessons to learn there.

00:31:05.380 | This is really the Lobner Prize, the Turing Test of our generation.

00:31:09.980 | I'm excited to see if anybody's able to solve the Alexa Prize.

00:31:13.580 | Again, Alexa Prize is your task with talking to a bot, and the measure of quality is the

00:31:23.140 | same as the Lobner Prize is just measuring how good was that conversation, but also the

00:31:27.500 | task is to try to continue the conversation for 20 minutes.

00:31:31.140 | If you try to talk to a bot today, and you have a choice to talk to a bot or go do something

00:31:37.100 | else, watch Netflix, you'll last probably less than 10 seconds.

00:31:42.900 | You'll be bored.

00:31:45.100 | The point is to continue trapping you in the conversation because you're enjoying it so

00:31:49.180 | much.

00:31:50.180 | The 20 minutes is that's a really nice benchmark for passing the spirit of what the Turing

00:31:56.460 | Test stood for.

00:31:58.620 | Alcos here from the Alexa Prize, from the Alcos bot.

00:32:03.100 | The difference in two kinds of conversations.

00:32:05.700 | Alcos says, "Have you been in Brazil?"

00:32:07.940 | The user says, "What is the population of Brazil?"

00:32:10.540 | Alcos says, "It is about 20 million."

00:32:13.460 | User says, "Well, okay."

00:32:15.980 | This is what happens a lot with, like I mentioned, multi-domain conversation is once you jump

00:32:21.540 | to a new domain, you stay there.

00:32:24.020 | Once you switch context, you stay there.

00:32:26.140 | The reality is you want to jump back and continue jumping around.

00:32:30.220 | In the second most successful conversation, "Have you been in Brazil?

00:32:34.620 | What is the population of Brazil?

00:32:36.420 | It is around 20 million.

00:32:38.220 | Anyway, I was saying, have you been in Brazil?"

00:32:42.340 | Jumping back in context.

00:32:43.980 | That's how conversation goes.

00:32:46.020 | Tangent to tangent and back.

00:32:48.060 | Quickly, there's been a lot of sequence to sequence kind of work using natural language

00:32:53.940 | to summarize a lot of applications.

00:32:57.460 | One of them, I cleared that I wanted to highlight from Technion that I find particularly interesting

00:33:03.620 | is the abstract syntax tree-based summarization of code.

00:33:10.780 | Modeling computer code, in this case, sadly, Java and C#, in trees, in syntax trees, and

00:33:18.820 | then operating on those trees to then do the summarization in text.

00:33:24.020 | Here an example of a basic power of two function on the bottom right in Java.

00:33:31.920 | The code to sec summarization says, "Get power of two."

00:33:36.980 | That's an exciting possibility of automated documentation of source code.

00:33:41.700 | I thought it was particularly interesting.

00:33:43.100 | The future there is bright.

00:33:44.620 | Okay, hopes for 2020 for natural language processing is reasoning.

00:33:49.300 | Common-sense reasoning becomes greater and greater part of the transformer-type language

00:33:53.940 | model work that we've seen in the deep learning world.

00:33:57.660 | Extending the context from hundreds or thousands of words to tens of thousands of words.

00:34:03.940 | Being able to read entire stories and maintain the context, which transformers, again, with

00:34:11.860 | ExcelNet, Transformer Excel is starting to be able to do, but we're still far away from

00:34:16.260 | that long-term, lifelong maintenance of context.

00:34:20.460 | Dialogue open domain dialogue, forever since Alan Turing to today is the dream of artificial

00:34:25.900 | intelligence being able to pass the Turing test.

00:34:29.660 | The dream of natural language model transformers are self-supervised learning.

00:34:39.380 | The dream of Yann LeCun is to, for these kinds of what previously were called unsupervised,

00:34:47.820 | but he's calling now self-supervised learning systems, to be able to sort of watch YouTube

00:34:53.380 | videos and from that start to form representation based on which you can understand the world.

00:34:59.660 | The hope for 2020 and beyond is to be able to transfer some of the success of transformers

00:35:05.700 | to the world of visual information, the world of video, for example.

00:35:13.900 | Deep RL and self-play.

00:35:15.780 | This has been an exciting year, continues to be an exciting time for reinforcement learning

00:35:22.780 | in games and robotics.

00:35:24.980 | So first, Dota 2 and OpenAI, an exceptionally popular competitive game, e-sports game that

00:35:33.300 | people compete, win millions of dollars with.

00:35:37.020 | So this is a lot of world-class professional players.

00:35:39.860 | So in 2018, OpenAI 5, this is a team play, tried their best at the international and

00:35:46.940 | lost and said that we're looking forward to pushing 5 to the next level, which they did

00:35:52.060 | in April 2018.

00:35:54.060 | They beat the 2018 world champions in 5 on 5 play.

00:36:01.260 | So the key there was compute, 8 times more training compute because the actual compute

00:36:09.940 | was already maxed out.

00:36:11.540 | The way they achieved the 8x is in time, simply training for longer.

00:36:15.420 | So the current version of OpenAI 5, as Jacob will talk about next Friday, has consumed

00:36:20.980 | 800 petaflop a second days and experienced about 45,000 years of Dota self-play over

00:36:27.300 | 10 real-time months.

00:36:28.940 | Again, behind a lot of the game systems talk about the, they use self-play so they play

00:36:33.500 | against each other.

00:36:34.500 | This is one of the most exciting concepts in deep learning, systems that learn by playing

00:36:39.540 | each other and incrementally improving in time.

00:36:43.560 | So starting from being terrible and getting better and better and better and better, and

00:36:47.580 | always being challenged by a slightly better opponent because of the natural process of

00:36:52.780 | self-play.

00:36:53.780 | That's a fascinating process.

00:36:56.060 | The 2019 version, the last version of OpenAI 5 has a 99.9 win rate versus the 2018 version.

00:37:05.220 | Then DeepMind also in parallel has been working and using self-play to solve some of these

00:37:11.780 | multi-agent games, which is a really difficult space when people have to collaborate as part

00:37:19.340 | of the competition.

00:37:20.340 | It's exceptionally difficult from the reinforcement learning perspective.

00:37:23.700 | So this is from raw pixels, solve the arena, capture the flag game, Quake three arena.

00:37:30.180 | One of the things I love just as a sort of side note about both OpenAI and DeepMind and

00:37:35.660 | general research and reinforcement learning, there will always be one or two paragraphs

00:37:39.580 | of philosophy.

00:37:41.520 | In this case from DeepMind, billions of people inhabit the planet, each with their own individual

00:37:46.900 | goals and actions, but still capable of coming together through teams, organizations, and

00:37:52.020 | societies in impressive displays of collective intelligence.

00:37:55.180 | This is a setting we call multi-agent learning.

00:37:58.380 | Many individual agents must act independently yet learn to interact and cooperate with other

00:38:03.300 | agent.

00:38:04.300 | This is immensely difficult problem because with co-adapting agent, the world is constantly

00:38:08.220 | changing.

00:38:09.220 | The fact that we 7 billion people on earth, people in this room, in families, in villages

00:38:16.780 | can collaborate while being for the most part self-interested agents is fascinating.

00:38:22.420 | One of my hopes actually for 2020 is to explore social behaviors that emerge in reinforcement

00:38:27.060 | learning agents and how those are echoed in real human to humans social systems.

00:38:34.540 | Okay, here's some visualizations.

00:38:36.700 | The agents automatically figure out as you see in other games, they figure out the concepts.

00:38:40.980 | So knowing very little, knowing nothing about the rules of the game, about the concepts

00:38:44.780 | of the game, about the strategy and the behaviors, they're able to figure it out.

00:38:48.260 | There's the T-SNE visualizations of the different states, important states and concepts in the

00:38:53.220 | game that this figures out and so on.

00:38:56.060 | Skipping ahead, automatic discovery of different behaviors.

00:38:59.540 | This happens in all the different games we talk about from Dota to Starcraft to Quake,

00:39:05.420 | the different strategies that it doesn't know about, it figures out automatically.

00:39:11.140 | And the really exciting work in terms of the multi-agent RL on the DeepMind side was the

00:39:17.660 | beating world-class players and achieving grand master level in a game I do know about,

00:39:23.820 | which is Starcraft.

00:39:25.220 | In December 2018, AlphaStar beat Mana, one of the world's strongest professional Starcraft

00:39:30.060 | players, but that was in a very constrained environment and it was a single race, I think

00:39:34.940 | Protoss.

00:39:39.140 | And in October 2019, AlphaStar reached grand master level by doing what we humans do.

00:39:44.020 | So using a camera, observing the game and playing as part of, against other humans.

00:39:50.780 | So this is not an artificial side system.

00:39:53.420 | This is doing exact same process humans would undertake and achieve grand master, which

00:39:57.420 | is the highest level.

00:39:58.820 | Okay, great.

00:39:59.820 | I encourage you to observe a lot of the interesting on their blog posts and videos of the different

00:40:05.940 | strategies that the RL agents are able to figure out.

00:40:10.780 | Here's a quote from one of the professional Starcraft players, and we see this with AlphaZero2

00:40:15.100 | in chess, is "AlphaStar is an intriguing unorthodox player, one with the reflexes and speed of

00:40:21.380 | the best pros, but strategies and style, they're entirely its own.

00:40:25.940 | The way AlphaStar was trained with agents competing against each other in a league has

00:40:30.880 | resulted in gameplay that's unimaginably unusual.

00:40:34.260 | It really makes you question how much of Starcraft's diverse possibilities pro players have really

00:40:39.460 | explored."

00:40:40.460 | And that's the really exciting thing about reinforcement learning agent in chess, in

00:40:44.060 | Go, in games, and hopefully simulated systems in the future that teach us, teach experts

00:40:51.000 | that think they understand the dynamics of a particular game, a particular simulation

00:40:57.020 | of new strategies, of new behaviors to study.

00:41:02.400 | That's one of the exciting applications from almost a psychology perspective that I'd love

00:41:06.720 | to see reinforcement learning push towards.

00:41:10.220 | And on the imperfect information game side, poker, in 2018, CMU, Noah Brown was able to

00:41:21.780 | beat head-to-head, no limit Texas Hold'em, and now team six player no limit Texas Hold'em

00:41:28.420 | against professional players.

00:41:31.160 | Many of the same results, many of the same approaches was self-play, iterated Monte Carlo,

00:41:38.460 | and there's a bunch of ideas in terms of the abstractions.

00:41:42.740 | So there's so many possibilities under the imperfect information that you have to form

00:41:47.180 | these bins of abstractions in both the action space in order to reduce the action space

00:41:52.580 | and the information abstraction space.

00:41:54.740 | So the probabilities of all the different hands that could possibly have and all the

00:41:59.220 | different hands that the betting strategies could possibly represent.

00:42:02.220 | And so you have to do this kind of course planning.

00:42:05.660 | So they use self-play to generate a course blueprint strategy that in real time, they

00:42:12.300 | then use Monte Carlo search to adjust as they play.

00:42:16.300 | Again, unlike the deep mind open eye approaches, very few, very minimal compute required, and

00:42:22.180 | they're able to achieve to beat world-class players.

00:42:26.180 | Again, I like this is getting quotes from the professional players after they get beaten.

00:42:33.380 | So Chris Ferguson, famous World Series of Poker player said, "Pleribus," that's the

00:42:38.820 | name of the agent, "is a very hard opponent to play against.

00:42:42.400 | It's really hard to pin him down on any kind of hand.

00:42:45.840 | He's also very good at making thin value bets on the river.

00:42:50.680 | He's very good at extracting value out of his good hands, sort of making bets without

00:42:56.200 | scaring off the opponent."

00:42:58.800 | Darin Elias said, "Its major strength is its ability to use mixed strategies.

00:43:04.080 | That's the same thing that humans try to do.

00:43:06.620 | It's a matter of execution for humans to do this in a perfectly random way and to do so

00:43:10.800 | consistently most people just can't."

00:43:14.680 | Then in the robotic space has been a lot of application reinforcement learning.

00:43:18.240 | One of the most exciting is the manipulation, sufficient manipulation to be able to solve

00:43:24.200 | the Rubik's Cube.

00:43:26.440 | Again this is learned through reinforcement learning.

00:43:30.600 | Again because self-plays in this context is not possible, they use automatic domain randomization,

00:43:35.440 | ADR.

00:43:36.440 | So they generate progressively more difficult environments for the hand.

00:43:39.760 | There's a giraffe head there you see.

00:43:42.080 | There's a lot of perturbations to the system, so they mess with it a lot.

00:43:45.480 | And then a lot of noise injected into the system to be able to teach the hand to manipulate

00:43:50.520 | the cube in order to then solve.

00:43:53.120 | The actual solution of figuring out how to go from this particular face to the solved

00:43:59.360 | cube is an obvious problem.

00:44:03.680 | This paper and this work is focused on the much more difficult learning to manipulate

00:44:10.480 | the cube.

00:44:11.480 | It's really exciting.

00:44:13.240 | Again a little philosophy as you would expect from OpenAI is they have this idea of emergent

00:44:18.400 | meta-learning.

00:44:20.160 | This idea that the capacity of the neural network that's learning this manipulation

00:44:25.520 | is constrained while the ADR, the automatic domain randomization, is progressively making

00:44:31.360 | harder and harder environment.

00:44:32.960 | So the capacity of the environment to be difficult is unconstrained.

00:44:36.920 | And because of that, there's an emergent self-optimization of the neural network to learn general concepts

00:44:44.880 | as opposed to memorize particular manipulations.

00:44:50.640 | The hope for me in the deep reinforcement learning space for 2020 is the continued application

00:45:00.080 | of robotics, even sort of legged robotics, but also robotic manipulation.

00:45:08.640 | Human behavior, the use of multi-agent self-plays I've mentioned to explore naturally emerging

00:45:12.820 | social behaviors, constructing simulations of social behavior, and seeing what kind of

00:45:18.720 | multi-human behavior emerges in self-play context.

00:45:23.040 | I think that's one of the nice, there are always, I hope there'll be like a reinforcement

00:45:29.480 | learning self-play psychology department one day, like where you use reinforcement learning

00:45:34.080 | to study, to reverse engineer human behavior and study it through that way.

00:45:40.480 | And again, in games, I'm not sure what the big challenges that remain, but I would love

00:45:44.900 | to see, to me at least, it's exciting to see learned solution to games, to self-play.

00:45:53.160 | Science and deep learning, I would say there's been a lot of really exciting developments

00:45:57.560 | here that deserve their own lecture.

00:45:59.120 | I'll mention just a few.

00:46:01.800 | Here from MIT in early 2018, but it sparked a lot of interest in 2019, follow on work,

00:46:08.560 | is the idea of the lottery ticket hypothesis.

00:46:11.480 | So this work showed that sub-networks, small sub-networks within the larger network are

00:46:19.960 | the ones that are doing all the thinking.

00:46:22.520 | The same results in accuracy can be achieved from a small sub-network from within a neural

00:46:28.200 | network.

00:46:29.200 | And they have a very simple process of arriving at a sub-network of randomly initializing

00:46:34.800 | a neural network.

00:46:37.160 | That's I guess the lottery ticket, train the network until it converges.

00:46:41.240 | This is an iterative process, prune the fraction of the network with low weights, reset the

00:46:46.520 | waste of the remaining network with the original initialization, the same lottery ticket, and

00:46:52.800 | then train again the pruned untrained network and continue this iteratively, continuously

00:47:02.640 | to arrive at a network that's much smaller using the same original initializations.

00:47:08.880 | This is fascinating that within these big networks, there's often a much smaller network

00:47:13.520 | that can achieve the same kind of accuracy.

00:47:16.600 | Now practically speaking, it's unclear what are the big takeaways there except the inspiring

00:47:22.200 | takeaway that there exist architectures that are much more efficient.

00:47:25.640 | So there's value in investing time in finding such networks.

00:47:30.440 | Then there is disentangled representations, which again deserves its own lecture, but

00:47:36.680 | here showing a 10 vector representation.

00:47:41.720 | And the goal is where each part of the vector can learn one particular concept about a data

00:47:46.200 | set.

00:47:47.200 | So the dream of unsupervised learning is you can learn compressed representations where

00:47:52.120 | every one thing is disentangled and you can learn some fundamental concept about the underlying

00:47:56.920 | data that can carry from data set to data set to data set.

00:48:01.800 | That's disentangled representation.

00:48:03.520 | There's theoretical work, best ICML paper in 2019 showing that that's impossible.

00:48:12.120 | So disentangled representations are impossible without inductive biases.

00:48:17.320 | And so the suggestion there is that the biases that you use should be made explicit as much

00:48:24.960 | as possible.

00:48:26.400 | The open problem is finding good inductive biases for unsupervised model selection that

00:48:31.200 | work across multiple data sets that we're actually interested in.

00:48:34.640 | A lot more papers, but one of the exciting is the double descent idea that's been extended

00:48:41.040 | and to the deep neural network context by OpenAI to explore the phenomena that as we

00:48:48.920 | increase the number of parameters in a neural network, the test error initially decreases,

00:48:53.320 | increases and just as the model is able to fit the training set undergoes a second descent.

00:48:58.160 | So decrease, increase, decrease.

00:49:00.840 | So there's this critical moment of time when the training set is just fit perfectly.

00:49:09.560 | Okay, and this is the OpenAI shows that it's applicable not just to model size, but also

00:49:14.400 | to training time and data set time.

00:49:16.320 | This is more like an open problem of why this is, trying to understand this and how to leverage

00:49:22.160 | it in optimizing training dynamics in neural networks.

00:49:26.600 | That's a, there's a lot of really interesting theoretical questions there.

00:49:30.480 | So my hope there for the science of deep learning in 2020 is to continue exploring the fundamentals

00:49:35.520 | of model selection, training dynamics, and the folks focused on the performance of the

00:49:41.000 | training in terms of memory and speed has worked on and the representation characteristics

00:49:45.640 | with respect to architecture characteristics.

00:49:48.000 | So a lot of the fundamental work there and understanding neural networks.

00:49:51.960 | Two areas that I had hold two sections on and papers, which is super exciting.

00:49:57.880 | My first love is graphs.

00:50:00.840 | So graph neural networks is a really exciting area of deep learning graph convolution neural

00:50:07.600 | networks as well for solving combinatorial problems and recommendation systems that are

00:50:11.920 | really useful in any kind of problem that is fundamentally can be modeled as a graph

00:50:17.120 | can be then solved or at least aided in by neural networks.

00:50:21.920 | There's a lot of exciting area there and Bayesian deep learning using Bayesian neural networks

00:50:28.760 | that's been for several years and exciting possibility.

00:50:31.840 | It's very difficult to train large Bayesian networks, but in the context that you can

00:50:37.360 | and it's useful small data sets providing uncertainty measurements in the predictions

00:50:43.000 | is extremely powerful capability of Bayesian nets, Bayesian neural networks and online

00:50:49.760 | incremental learning.

00:50:50.800 | These neural networks release it.

00:50:52.160 | There's a lot of really good papers there.

00:50:54.000 | It's exciting.

00:50:55.000 | Okay.

00:50:56.000 | Autonomous vehicles.

00:50:57.000 | Oh boy.

00:50:58.000 | So let me try to use as few sentences as possible to describe this section of a few slides.

00:51:04.440 | It is one of the most exciting areas of applications of AI and learning in the real world today.

00:51:13.280 | And I think it's the way that artificial intelligence, it is the place where artificial intelligence

00:51:18.240 | systems touch human beings that don't know anything about artificial intelligence.

00:51:22.880 | The most hundreds of thousands soon millions of cars will be interacting with human beings,

00:51:29.000 | robots really.

00:51:30.000 | So this is a really exciting area and really difficult problem.

00:51:33.800 | And there's two approaches.

00:51:35.280 | One is level two where the human is fundamentally responsible for the supervision of the AI

00:51:40.600 | system and level four, or at least the dream is where the AI system is responsible for

00:51:46.040 | the actions and the human does not need to be a supervisor.

00:51:49.960 | Okay.

00:51:50.960 | So level two companies represent each of these approaches that are sort of leading the way.

00:51:56.080 | Waymo in October, 2018, 10 million miles on road.

00:52:01.320 | Today this year, they've done 20 million miles in simulation, 10 billion miles and a lot.

00:52:08.880 | I got a chance to visit them out in Arizona.

00:52:11.400 | They're doing a lot of really exciting work and they're obsessed with testing.

00:52:15.080 | So the kind of testing they're doing is incredible.

00:52:17.520 | 20,000 classes of structured tests of putting the system through all kinds of tests that

00:52:23.960 | the engineers can think through and that appear in the real world.

00:52:28.280 | And they've initiated testing on road with real consumers without a safety driver, which

00:52:36.440 | if you don't know what that is, that means the car is truly responsible.

00:52:40.560 | There's no human catch.

00:52:43.200 | The exciting thing is that there is 700,000, 800,000 Tesla autopilot systems.

00:52:53.640 | That means there's these systems that are human supervised.

00:52:56.280 | They're using a multi-headed neural network, multitask neural network to perceive, predict

00:53:06.200 | and act in this world.

00:53:09.680 | That's a really exciting real world deployment, large scale of neural networks.

00:53:15.640 | Is a fundamentally deep learning system.

00:53:17.840 | Unlike Waymo, which is deep learning is the icing on the cake for Tesla, deep learning

00:53:24.600 | is the cake.

00:53:27.160 | It's at the core of the perception and the action that the system performs.

00:53:32.960 | They have to date done over 2 billion miles estimated, and that continues to quickly grow.

00:53:39.320 | I'll briefly mention, which I think is a super exciting idea in all applications of machine

00:53:45.440 | learning in the real world, which is online.

00:53:49.360 | So iterative learning, active learning, Andrej Karpathy, who's the head of autopilot causes

00:53:55.240 | this, the data engine.

00:53:57.200 | It's this iterative process of having a neural network performing the task, discovering the

00:54:00.880 | edge cases, searching for other edge cases that are similar, and then retraining the

00:54:06.240 | network, annotating the edge cases and then retraining there and continuously doing this

00:54:09.400 | loop.

00:54:10.400 | This is what every single company that's using machine learning seriously is doing.

00:54:14.000 | Very little publications on this space and active learning, but this is the fundamental

00:54:17.720 | problem of machine learning.

00:54:19.120 | It's not to create a brilliant neural network, it's to create a dumb neural network that

00:54:23.880 | continuously learns to improve until it's brilliant.

00:54:28.720 | And that process is especially interesting when you take it outside of single task learning.

00:54:33.440 | So most papers are written on single task learning.

00:54:35.600 | You take whatever benchmark, here in the case of driving, it's object detection, landmark

00:54:40.400 | detection, drivable area, trajectory generation, right?

00:54:45.920 | All those have benchmarks and you can have separate neural networks for them.

00:54:50.280 | That's a single task.

00:54:51.280 | But combining to use a single neural network that performs all those tasks together, that's

00:54:55.960 | the fascinating challenge where you're reusing parts of the neural network to learn things

00:54:59.920 | that are coupled and then to learn things that are completely independent.

00:55:03.560 | And doing the continuous active learning loop.

00:55:07.840 | There inside companies, in the case of Tesla and Waymo in general, it's exciting to have

00:55:14.200 | people, these are actual human beings that are responsible for these particular tasks.

00:55:18.760 | They become experts of particular perception tasks, experts of particular planning tasks

00:55:23.560 | and so on.

00:55:24.560 | And so the job of that expert is both to train the neural network and to discover the edge

00:55:28.920 | cases which maximize the improvement of the network.

00:55:31.720 | That's where the human expertise comes in a lot.

00:55:36.040 | And there's a lot of debate.

00:55:37.520 | It's an open question about which kind of system would be, which kind of approach would

00:55:42.200 | be successful.

00:55:43.200 | A fundamentally learning based approach as is with the level two, with the Tesla autopilot

00:55:48.320 | system that's learning all the different tasks that are involved with driving.

00:55:55.180 | And as it gets better and better and better, less and less human supervision is required.

00:55:58.800 | The pro of that approach is the camera based systems have the highest resolution so that

00:56:04.480 | it's very amenable to learning.

00:56:06.240 | But the con is that it requires a lot of data, a huge amount of data.

00:56:11.160 | And nobody knows how much data yet.

00:56:14.480 | The other con is human psychology is the driver behavior that the human must continue to remain

00:56:20.800 | vigilant.

00:56:21.800 | On the level four approach that leverages besides cameras and radar and so on also leverages

00:56:28.800 | LIDAR map.

00:56:31.080 | The pros that it's much more consistent, reliable, explainable system.

00:56:36.840 | So the detection, the accuracy of the detection, the depth estimation, the detection of different

00:56:42.520 | objects is much higher, accurate with less data.

00:56:46.920 | The cons is it's expensive, at least for now.

00:56:49.980 | It's less amenable to learning methods because much fewer data, lower resolution data and

00:56:57.080 | must require at least for now, some fallback, whether that's the safety driver or teleoperation.

00:57:04.400 | The open questions for the deep learning level two Tesla autopilot approach is how hard is

00:57:10.280 | driving?

00:57:11.280 | This is actually the open question for most disciplines in artificial intelligence.

00:57:15.180 | How difficult is driving?

00:57:16.500 | How many edge cases does driving have?

00:57:18.960 | And that can we learn to generalize over those edge cases without solving the common sense

00:57:23.800 | reasoning problem?

00:57:24.800 | This kind of, it's kind of a task without solving the human level artificial intelligence

00:57:28.520 | problem.

00:57:29.560 | And that means perception.

00:57:31.120 | How hard is perception detection, intention, modeling, human mental model, modeling the

00:57:37.400 | trajectory prediction, and then the action side, the game theoretic action side of balancing.

00:57:43.080 | Like I mentioned, fun and enjoyability with the safety of the systems, because these are

00:57:48.400 | life critical systems and human supervision, the vigilance side, how good can autopilot

00:57:54.320 | get before visuals decrements significantly and people fall asleep, become distracted,

00:57:58.920 | start watching movies, so on and so on.

00:58:00.720 | The things that people naturally do.

00:58:02.480 | The open question is how good can autopilot get before that becomes a serious problem?

00:58:07.720 | And if that decrement nullifies the safety benefit of the use of autopilot, which is

00:58:14.800 | autopilot AI system, when the sensors are working well is perfectly vigilant.

00:58:21.800 | The AI is always paying attention.

00:58:26.760 | The open questions for the LIDAR based, the level four, the way more approaches when we

00:58:32.140 | have maps, LIDAR and geo-fenced routes that are taken, how difficult is driving?

00:58:39.120 | The traditional approach to robotics, the, from the DARPA challenge to today for most

00:58:43.600 | autonomous vehicle companies is to do HD maps, to use LIDAR for really accurate localization

00:58:50.360 | together with GPS.

00:58:52.000 | And then the perception problem becomes the icing on the cake because you already have

00:58:56.240 | a really good sense of where you are with obstacles in the scene.

00:58:59.320 | And the perception is not a safety critical task, but a task of interpreting the environment

00:59:04.600 | further.

00:59:05.600 | So you have more, it's naturally by nature already safer.

00:59:13.360 | But how difficult is nevertheless is that problem?

00:59:15.800 | If perception is the hard problem, then the LIDAR based approaches is nice.

00:59:21.920 | If action is the hard problem, then both Tesla and Waymo have to solve the action problem

00:59:26.880 | without the sensors don't matter there.

00:59:29.640 | It's the difficult problem, the planning, the game theoretic, the human, the modeling

00:59:36.020 | of mental models and the intentions of other human beings, the pedestrians and the cyclists

00:59:40.760 | is the hard problem.

00:59:42.640 | And then the other side, the 10 billion miles of simulation, the open problem from reinforcement

00:59:47.160 | learning, deep learning in general is how much can we learn from simulation?

00:59:50.620 | How much of that knowledge can we transfer to then read the real world systems?

00:59:56.020 | My hope in the autonomous vehicle space, AI assisted driving space is to see more applied

01:00:02.980 | deep learning innovation.

01:00:04.140 | Like I mentioned, these are really exciting areas, at least to me of active learning,

01:00:08.440 | multitask learning and lifelong learning, online learning, iterative learning.

01:00:12.560 | There's a million terms for it, but basically continually learning and then the multitask

01:00:18.200 | learning to solve multiple problems.

01:00:21.560 | Over the air updates, I would love to see in terms of the autonomous vehicle space.

01:00:25.920 | This is common for, this is a prerequisite for online learning.

01:00:31.080 | If you want a system that continuously improves some data, you want to be able to deploy new

01:00:35.020 | versions of that system.

01:00:37.120 | Autonomous is one of the only vehicles that I'm aware of in the level two space that's

01:00:41.240 | deploying software updates regularly and built an infrastructure to deploy those updates.

01:00:46.760 | So updating neural networks, that to me seems like a prerequisite for solving the problem

01:00:52.760 | of autonomy in the level two space, any space is deploy updates.

01:00:59.440 | And for research purposes, public datasets continue.

01:01:02.900 | There's already a few public datasets of edge cases, but I'd love to continue seeing that

01:01:06.760 | from automotive companies and autonomous vehicle companies and simulators.

01:01:11.760 | Carla, NVIDIA, DRIVE Constellation, Voyage, Deep Drive.

01:01:14.640 | There's a bunch of simulators coming out that are allowing people to experiment with perception,

01:01:20.720 | with planning, with reinforcement learning algorithms.

01:01:23.760 | I'd love to see more of that and less hype, of course, less hype.

01:01:27.800 | One of the most overhyped spaces besides sort of AI generally is autonomous vehicles.

01:01:33.120 | And I'd love to see real balanced, nuanced, in-depth reporting by journalists and companies

01:01:39.600 | on successes and challenges of autonomous driving.

01:01:42.640 | If we skip any section, it would be politics, but maybe briefly mention, somebody said Andrew

01:01:50.880 | Yang.

01:01:54.360 | So it's exciting for me to see, exciting and funny and awkward to see artificial intelligence

01:02:01.560 | discussed in politics.

01:02:03.200 | So one of the presidential candidates discussing artificial intelligence, awkwardly, sort of

01:02:08.080 | there's interesting ideas, but there's still a lack of understanding of fundamentals of

01:02:11.080 | artificial intelligence.

01:02:12.080 | There's a lot of important issues, but he's bringing artificial intelligence to the public

01:02:16.280 | discourse.

01:02:17.640 | That's nice to see, but it is the early days.

01:02:21.800 | And so as a community, that informs me that we need to communicate better about the limitation

01:02:26.000 | capabilities of artificial intelligence and automation broadly.

01:02:29.320 | The American initiative, AI initiative was launched this year, which is our government's

01:02:34.160 | best attempt to provide ideas and regulations about what does the future of artificial intelligence

01:02:40.280 | look like in our country.

01:02:41.640 | Again, awkward, but important to have these early developments, early ideas from the federal

01:02:49.800 | government about what are the dangers and what are the hopes, the funding and the education

01:02:58.040 | required to build a successful infrastructure for artificial intelligence.

01:03:03.840 | This is the fun part.

01:03:05.120 | There's a lot of tech companies being brought before government.

01:03:08.040 | It's really interesting in terms of power.

01:03:12.560 | Some of the most powerful people in our world today are the leaders of tech companies.

01:03:17.840 | And the fundamentals of what the tech companies work on is artificial intelligence systems,

01:03:24.520 | really recommendation systems, advertisement, discovery from Twitter to Facebook to YouTube,

01:03:33.720 | these are recommendation systems.

01:03:35.560 | And all of them are now fundamentally based on deep learning algorithms.

01:03:39.160 | So you have these incredibly rich, powerful companies that are using deep learning coming

01:03:44.640 | before government that's trying to see, awkwardly trying to see how can we regulate.

01:03:49.920 | And it's, I think the role of the ad community broadly to inform the public and inform government

01:03:56.880 | of how we talk about, how we think about these ideas.

01:04:01.560 | And also I believe it's the role of companies to publish more.

01:04:06.200 | There's been very little published on the details of recommendation systems behind Twitter,

01:04:11.520 | Facebook, YouTube, Google.

01:04:14.880 | So all those systems, there's very little that's published.

01:04:18.440 | Perhaps it's understandable why, but nevertheless, as we consider the ethical implications of

01:04:23.680 | these algorithms, there needs to be more publication.

01:04:26.160 | So here's just a harmless example from DeepMind talking about the recommendation system behind

01:04:32.960 | the Play Store app discovery.

01:04:35.180 | So there, there's a bunch of discussion about the kind of neural net that's being used to

01:04:41.160 | propose the candidate generation.

01:04:44.000 | So this is after you install a few apps, the generation of the candidate, it's shows you

01:04:48.920 | ranked the next app that you're likely to enjoy installing.

01:04:52.960 | And so there they tried LSTM and transformers and then narrowed it down to a more efficient

01:04:59.160 | model that's being able to run fast.

01:05:01.280 | That's a, that's a, that's an attention model.

01:05:05.140 | And then there's some, again, harmless, de-biasing, harmless in terms of topics.

01:05:11.800 | The model learns to bias in favor of the apps that are shown and thus installed more often

01:05:18.360 | as opposed to the ones you want.

01:05:19.960 | So there's some waiting to adjust for the biasing towards the apps that are popular

01:05:25.280 | to allow the possibility of you installing apps that are less popular.

01:05:28.900 | So that kind of process and publishing in and discussing in public, I think is really

01:05:32.920 | important and I would love to see more of that.

01:05:36.520 | So my hope in this, in the politics space and the public discourse space for 2020 is

01:05:44.120 | less fear of AI and more discourse between government and experts on topics of privacy,

01:05:52.320 | cybersecurity, and so on.

01:05:53.600 | And then transparency and recommender systems.

01:05:55.680 | I think the most exciting, the most powerful artificial intelligence system space for the

01:06:02.080 | next couple of decades is recommendation systems.

01:06:05.200 | Very little talked about, it seems like, but they're going to have the biggest impact on

01:06:09.400 | our society because they affect how the information we see, how we learn, what we think, how we

01:06:16.600 | communicate.

01:06:18.420 | These algorithms are controlling us.

01:06:22.540 | And we have to really think deeply as engineers of how to speak up and think about their,

01:06:32.960 | their societal implications.

01:06:35.180 | Not just in terms of bias and so on, which are sort of ethical considerations, which

01:06:39.440 | are really important, but stuff that's like the elephant in the room that's hidden, which

01:06:44.520 | is controlling how we think, how we see the world, the moral system under which we operate.

01:06:52.360 | Quickly to mention and wrapping up with a few minutes of questions, if there are any,

01:06:59.280 | is the deep learning courses this year, before the last few years has been a lot of incredible

01:07:06.680 | courses on deep learning, on reinforcement learning.

01:07:09.600 | What I would very much recommend for people is the fast AI course from Jeremy Howard,

01:07:17.240 | which uses their wrapper around PyTorch.

01:07:20.040 | It's to me, the best introduction to deep learning for people who are here or might

01:07:24.460 | be listening elsewhere, are thinking about learning more about deep learning.

01:07:28.520 | That's that, that is the, to me, the best course.

01:07:31.760 | Also paid, but Andrew Ang, everybody loves Andrew Ang, is the deep learning AI course,

01:07:39.000 | Sarah course on deep learning is, is, is excellent for, especially for complete beginner, for

01:07:44.040 | sort of beginners.

01:07:46.320 | And then Stanford has two excellent courses on visual recognition.

01:07:52.280 | So convolution neural nets, originally taught by Andrew Karpathy and natural language processing,

01:07:57.920 | excellent courses.

01:07:58.920 | And of course, here at MIT, there's a bunch of courses, especially on the fundamentals,

01:08:03.800 | on the mathematics, linear algebra, statistics.

01:08:07.240 | And I have a few lectures up online that you should never watch.

01:08:11.720 | Then on the reinforcement learning side, David Silver is one of the greatest people in understanding

01:08:17.000 | reinforcement learning from deep mind.

01:08:18.400 | He has a great course, an introduction to reinforcement learning, spinning up and deeper

01:08:22.200 | enforcement learning from open AI, I highly recommend.

01:08:25.160 | Here just over the slides that I'll share online, there's been, there's a lot of tutorials.

01:08:30.080 | One of my favorite lists of tutorials, which is, I believe the best way to learn machine

01:08:34.040 | learning, deep learning, natural language processing in general is, it's just code.

01:08:38.960 | Just build it yourself, build the models oftentimes from scratch.

01:08:41.920 | Here's a list of tutorials with that link over 200 tutorials on topics from deep RL

01:08:47.560 | to optimization to back prop, LSTMs, convolutional recurrent neural networks, everything.

01:08:54.080 | Over 200 of the best machine learning NLP and Python tutorials by Robbie Allen.

01:08:58.960 | You can Google that, or you can click the link.

01:09:01.640 | I love it, highly recommend.

01:09:03.160 | The three books I would recommend, of course, the deep learning book by Yoshua Bengio and

01:09:08.760 | Ian Goodfellow and Aaron Corville.

01:09:13.600 | That's more sort of the fundamental thinking about from philosophy to the specific techniques

01:09:19.440 | of the deep learning and the practical grok in deep learning, which Andrew Trask will

01:09:24.600 | be here Wednesday.

01:09:26.200 | His book, Grok in Deep Learning, I think is the best for beginners book on deep learning.

01:09:30.440 | I love it.

01:09:31.440 | He implements everything from scratch.

01:09:33.080 | It's extremely accessible.

01:09:34.560 | 2019 I think it was published, maybe 18, but I love it.

01:09:38.760 | And then Francois Chollet, the best book on Keras and TensorFlow and really deep learning

01:09:46.720 | as well as deep learning with Python, although you shouldn't buy it, I think, because he

01:09:51.640 | is supposed to come up with version two, which I think will cover TensorFlow 2.0.

01:09:55.440 | It'll be an excellent book.

01:09:58.160 | And when he's here Monday, you should torture him and tell him to finish writing.

01:10:01.880 | He was supposed to finish writing in 2019.

01:10:05.400 | My general hopes, as I mentioned, for 2020 is I'd love to see common sense reasoning

01:10:10.240 | enter the, not necessarily enter the world of deep learning, but be a part of artificial

01:10:16.120 | intelligence and the problems that people tackle.

01:10:19.360 | As I've been harboring, active learning is to me is the most important aspect of real

01:10:24.040 | world application of deep learning.

01:10:26.200 | There's not enough research.

01:10:27.200 | There should be way more research.

01:10:28.360 | I'd love to see active learning, lifelong learning.

01:10:31.320 | That's what we all do as human beings.

01:10:33.160 | That's what AI systems need to do.

01:10:34.920 | Continually learn from their mistakes over time.

01:10:38.560 | Figure out dumb, become brilliant over time.

01:10:41.720 | Open domain conversation with the Alexa prize.

01:10:45.520 | I would love to see breakthroughs there.

01:10:48.240 | Alexa folks thinks we're still two or three decades away, but that's what everybody says

01:10:53.880 | before the breakthrough.

01:10:55.320 | So I'm excited to see if there's any brilliant grad students that come up with something

01:10:59.960 | there.

01:11:01.440 | Applications in autonomous vehicles and medical space, algorithmic ethics.

01:11:05.000 | Of course, ethics has been a lot of excellent work.

01:11:08.240 | In fairness, privacy and so on, robotics.

01:11:13.080 | And as I said, recommendation systems, the most important in terms of impact part of

01:11:18.200 | artificial intelligence systems.

01:11:20.080 | I mentioned soup in terms of progress.

01:11:23.640 | There's been a little bit of tension, a little bit of love online in terms of deep learning.

01:11:28.640 | So I just wanted to say that the kind of criticism and skepticism about the limitations of deep

01:11:35.160 | learning are really healthy in moderation.

01:11:39.920 | Jeff Hinton, one of the three people to receive the Turing Award, as many people know, has

01:11:45.600 | said that the future depends on some graduate student who is deeply suspicious of everything

01:11:49.960 | I have said.

01:11:51.240 | So that suspicion, skepticism is essential, but in moderation, just a little bit.

01:11:57.600 | The more important thing is perseverance, which is what Jeffrey Hinton and the others

01:12:03.280 | have had through the winters of believing in neural nets and an open-mindedness for

01:12:09.040 | returning to the world of symbolic AI, of expert systems, of complexity and cellular

01:12:15.320 | automata, of old ideas in AI and bringing them back and see if there's ideas there.

01:12:20.080 | And of course, you have to have a little bit of crazy.

01:12:23.120 | Nobody ever achieves something brilliant without being a little bit crazy.

01:12:26.600 | And the most important thing is a lot of hard work.

01:12:30.680 | It's not the cool thing these days, but hard work is everything.

01:12:34.880 | I like what JFK said about us going to the moon.

01:12:39.880 | Us.

01:12:40.880 | I was born in the Soviet Union.

01:12:43.400 | See how I conveniently just said us?

01:12:48.440 | Going to the moon is we do these things not because they're easy, but because they're

01:12:53.560 | hard.

01:12:54.560 | And I think that artificial intelligence is one of the hardest and most exciting problems

01:12:59.640 | there before us.

01:13:01.520 | So with that, I'd like to thank you and see if there's any questions.

01:13:11.480 | Back in the 1980s, parallel distributing processing books came out.

01:13:14.840 | They had most of the stuff in it back then.

01:13:17.240 | What's your take on the roadblocks, the most important roadblocks, apart from maybe funding?

01:13:22.800 | I think fundamentally, I mean, they're well known as limitations, is that they're really

01:13:31.240 | inefficient at learning.

01:13:35.240 | And they're not-- so they're really good at extracting representations from raw data,

01:13:41.620 | but not good at learning knowledge bases of accumulating knowledge over time.

01:13:50.240 | That's the fundamental limitation.

01:13:51.680 | Computer systems are really good at accumulating knowledge, but very bad at doing that in an

01:13:58.120 | automated way.

01:14:00.040 | Symbolic AI.

01:14:02.120 | I don't know how to overcome.

01:14:04.880 | A lot of people say there's hybrid approaches.

01:14:07.280 | I believe more data, bigger networks.

01:14:10.960 | And better selection of data will take us a lot farther.

01:14:14.280 | Hello, Lex.

01:14:16.720 | I'm wondering if you recall what was the initial spark or inspiration that drove you

01:14:22.920 | towards work in AI?

01:14:24.520 | Was it when you were pretty young, or was it in more recent years?

01:14:29.600 | So I wanted to become a psychiatrist.

01:14:31.040 | I wanted to-- I thought of it as kind of engineering the human mind by sort of manipulating it.

01:14:39.280 | That's what I thought of psychiatry is by using words to sort of explore the depths

01:14:45.280 | of the mind and be able to adjust it.

01:14:47.080 | But then I realized that psychiatry can't actually do that.

01:14:50.720 | And modern psychiatry is more about sort of bioengineering, is drugs.

01:14:55.320 | And so I thought that the way to really explore the engineering of the mind is the other side,

01:15:01.960 | is to build it.

01:15:04.800 | And that's also when C++ really became the cool, hot thing.

01:15:10.240 | So I learned to program at 12 and then never looked back, hundreds of thousands of lines

01:15:15.400 | later.

01:15:16.400 | Just I love program.

01:15:17.400 | I love building.

01:15:18.400 | And that, to me, is the best way to understand the mind is to build it.

01:15:22.400 | Speaking of building mind, do you personally think that machines will ever be able to think?

01:15:29.080 | And the second question, will they ever be able to feel emotions?

01:15:32.760 | 100% yes.

01:15:34.560 | 100% they'll be able to think and they'll be able to feel emotions.

01:15:41.080 | So those concepts of thought and feeling are human concepts.

01:15:49.400 | And to me, they'll be able to fake it.

01:15:57.400 | Therefore, they'll be able to do it.

01:16:04.480 | I've been playing with Roombas a lot recently, Roomba vacuum cleaners.

01:16:11.000 | And so I've now started having Roombas scream.

01:16:16.000 | There's moaning in pain.

01:16:19.320 | And they became-- I feel like they're having emotions.

01:16:23.600 | So the faking creates the emotion.

01:16:27.440 | Yeah, so the display of emotion is emotion to me.

01:16:33.120 | And then the display of thought is thought.

01:16:35.400 | I guess that's the sort of everything else is-- everything else is impossible to pin

01:16:46.240 | down.

01:16:47.760 | I'm asking, so what about the ethical aspects of it?

01:16:52.080 | I'm asking because I was born in the Soviet Union as well.

01:16:55.520 | And one of my favorite recent books is Victor Pilevin's I Fuck.

01:16:58.920 | And it's about AI feeling emotions and suffering from it.

01:17:03.080 | So I don't know if you've read that book.

01:17:05.960 | What do you think about AI feeling emotions in that context or in general ethical aspects?

01:17:12.880 | Yeah, it's a really difficult question to answer.

01:17:15.920 | Yes, I believe AI will suffer.

01:17:18.160 | And it's unethical to torture AI.

01:17:20.720 | But I believe suffering exists in the eye of the observer.

01:17:26.720 | Sort of like if a tree falls and nobody's around to see it, it never suffered.

01:17:35.880 | It's us humans that see the suffering in the tree, in the animal, in our fellow humans.

01:17:41.300 | And sort of in that sense, the first time a programmer with a straight face delivers

01:17:47.320 | a product that says it's suffering is the first time it becomes unethical to torture

01:17:54.280 | AI systems.

01:17:56.920 | And we can do that today.

01:17:58.280 | I already built the Roombas.

01:18:00.120 | They won't sell currently.

01:18:02.440 | But I think the first time a Roomba says, please don't hurt me, that's when we start

01:18:09.840 | to have serious conversations about the ethics.

01:18:13.320 | And it sounds ridiculous.

01:18:14.760 | I'm glad this is being recorded because it won't be ridiculous in just a few years.

01:18:21.280 | Is reinforcement learning a good candidate for achieving general artificial intelligence?

01:18:30.080 | Are there any other good candidates around?

01:18:33.000 | So to me, the answer is no.

01:18:35.460 | But it can teach us some valuable gaps that can be filled by other methods.

01:18:43.120 | So I believe that simulation is different than the real world.

01:18:45.720 | So if you could simulate the real world, then deep RL, any kind of reinforcement learning

01:18:52.880 | with deep representations would be able to achieve something incredible.

01:18:56.240 | But to me, simulation is very different than the real world.

01:18:58.080 | So you have to interact in the real world.

01:19:00.160 | And there, you have to be much more efficient with learning.

01:19:02.480 | And to be more efficient with learning, you have to have ability to automatically construct

01:19:07.400 | common sense.

01:19:09.320 | Like common sense reasoning seems to include a huge amount of information that's accumulated

01:19:18.000 | over time.

01:19:20.040 | And that feels more like programs than functions.

01:19:25.320 | I like how Elias Esquivir talks about deep learning learns functions, approximators.

01:19:31.520 | Deep RL learns an approximator for policy or whatever, but not programs.

01:19:36.600 | It's not learning a thing that's able to sort of-- that's essentially what reasoning is

01:19:43.480 | is a program.

01:19:44.480 | It's not a function.

01:19:46.520 | So I think no.

01:19:49.160 | But it'll continue to, one, inspire us and, two, inform us about where the true gaps are.

01:19:54.360 | I think the ability to-- but I'm so human-centric.

01:19:58.360 | But I think the approach of being able to take knowledge and put it together, sort of

01:20:04.480 | building it to more and more complicated pieces of information, concepts, being able to reason

01:20:09.760 | in that way.

01:20:11.160 | There's a lot of methodologies that old school sort of-- that falls under the ideas of symbolic

01:20:16.640 | AI, of doing that kind of logic reasoning, accumulating knowledge bases.

01:20:21.580 | That's going to be an essential part of general intelligence.

01:20:24.880 | But also the essential part of general intelligence is the Roomba that says, I'm intelligent,

01:20:31.120 | F you, if you don't believe me.

01:20:34.440 | Like a very confident-- because right now, Alexa is very nervous, like, oh, what can

01:20:40.120 | I do for you?

01:20:42.360 | But once Alexa says, is upset that you would turn her off or treat her like a servant or

01:20:57.600 | say that she's not intelligent, that's when intelligence starts emerging.

01:21:02.000 | Because I think humans are pretty dumb.

01:21:03.560 | And what-- in general, we're all-- like, intelligence is a very kind of relative human construct

01:21:11.280 | that we've kind of convinced each other of.

01:21:13.760 | And once AI systems are also playing that game of creating constructs and that human

01:21:22.400 | communication, that's going to be important.

01:21:24.080 | But of course, for that, you still need to have pretty good, witty conversation.

01:21:28.680 | And for that, you need to do the symbolic AI, I think.

01:21:31.680 | I'm wondering about the autonomous vehicles, whether they are responsive to environmental

01:21:36.440 | sounds.

01:21:37.440 | I mean, if I notice an autonomous vehicle driving erratically, will it respond to my

01:21:41.080 | beep?

01:21:42.920 | That's a really interesting question.

01:21:43.920 | As far as I know, no.

01:21:46.120 | I think Waymo hinted that they look at sound a little bit.

01:21:49.880 | I think they should.

01:21:50.920 | So there's a lot of stuff that comes from audio that's really interesting.

01:21:53.840 | The sort of-- Waymo have said that they use audio for sirens.

01:21:59.760 | So detecting sirens from far away.

01:22:02.760 | I think audio is a lot of interesting information.

01:22:06.880 | The sound that the tires make on different kinds of roads is very interesting.

01:22:13.480 | We kind of-- we use that information ourselves, too, depending on kind of like off-road.

01:22:19.600 | Wet road, when it's not raining, sounds different than dry road.

01:22:23.920 | There's a lot of little subtle information.

01:22:26.720 | Pedestrians yelling and that kind of stuff.

01:22:28.960 | It's actually very difficult to know how much you get from audio.

01:22:33.000 | Most robotics folks think that audio is useless.

01:22:36.720 | I'm a little skeptical.

01:22:39.720 | But nobody's been able to identify why audio might be useful.

01:22:43.620 | So I have two questions.

01:22:45.920 | My first is, what do you think is the ultimate sort of endpoint for super machine intelligence?

01:22:54.200 | Like, will we sort of be relegated to some obscure part of the Earth like we've done

01:22:59.560 | to the next primates, the next intelligent primates?

01:23:03.040 | And my second question is, should we have equal rights for beings made out of silicon

01:23:09.760 | versus carbon, for example?

01:23:13.120 | Like robots or, you know?

01:23:15.720 | Separate rights or same rights?

01:23:17.160 | Like equal rights with humans.

01:23:19.360 | Yeah.

01:23:21.240 | So the future of super intelligence, I think I have much less worry-- I see much fewer

01:23:28.000 | paths to AI, AGI systems killing humans than I do for AGI systems living among us.

01:23:35.320 | So I think I see exciting or not so exciting but not harmful futures.

01:23:44.880 | I think it's very difficult to create AI systems that will kill people, that aren't

01:23:51.880 | like literally weapons of war.

01:23:54.920 | It'll always be people killing people.

01:23:57.680 | Like the things we should be worried about is other people.

01:24:00.520 | That's the fundamental.

01:24:02.520 | So there's a lot of ways.

01:24:03.520 | Yeah, nuclear weapons.

01:24:04.520 | There's a lot of existential threats to our society that are fundamentally human at the

01:24:08.440 | core.

01:24:09.440 | And I think AI might be tools of that, but there'll be also tools to defend against that.

01:24:15.920 | I also see AI proliferating as companions.

01:24:19.520 | I think companionship will be a really interesting-- like we will more and more live as we already

01:24:26.040 | do in the digital world.

01:24:27.520 | Like you have an identity on Twitter and Instagram, especially if it's anonymous or something.

01:24:32.360 | You have this identity you've created.

01:24:34.460 | And that will continue growing more and more, especially for people born now.

01:24:38.280 | But it's kind of this artificial identity that we live much more in the digital space.

01:24:43.240 | And in that digital space, as opposed to the physical space, is where AI can thrive much

01:24:47.480 | more currently.

01:24:49.480 | It'll thrive there first.

01:24:51.320 | So we'll live in a world with a lot of intelligent first assistants, but also just intelligent

01:24:56.240 | agents.

01:24:57.520 | And I do believe they should have rights.

01:25:05.420 | And in this contentious time of people, groups fighting for rights, I feel really bad saying

01:25:13.000 | they should have equal rights.

01:25:15.680 | But I believe that.

01:25:18.160 | I've talked to-- if you read the work of Peter Singer, of looking-- like my favorite food

01:25:27.160 | is steak.

01:25:28.160 | I love meat.

01:25:29.160 | But I also feel horrible about the torture of animals.

01:25:35.200 | And that's the same kind of-- to me, the way our society thinks about animals is a very

01:25:42.140 | similar way we should be thinking about robots, or we will be thinking about robots in, I

01:25:48.000 | would say, about 20 years.

01:25:51.440 | One final question.

01:25:53.400 | Will they become our masters?

01:25:56.400 | No.

01:25:59.880 | They will not be our masters.

01:26:01.200 | What I'm really worried about is who will become our masters are owners of large tech

01:26:09.020 | companies who use these tools to control human beings, first unintentionally and then intentionally.

01:26:19.300 | So we need to make sure that we democratize AI.

01:26:24.560 | It's the same kind of thing that we did with government.

01:26:29.600 | We make sure that we, at the heads of tech companies-- maybe people in this room will

01:26:35.320 | be heads of tech companies one day-- we have people like George Washington who relinquished

01:26:42.200 | power at the founding of this country.

01:26:45.560 | Forget all the other horrible things he did.

01:26:47.080 | But he relinquished power, as opposed to Stalin and all the other horrible human beings who

01:26:55.160 | have sought instead absolute power, which will be the 21st century.

01:27:02.360 | AI will be the tools of power in the hands of 25-year-old nerds.

01:27:08.920 | We should be very careful about that future.

01:27:11.800 | So the humans will become our masters, not the AI.

01:27:14.160 | AI will save us.

01:27:17.040 | So on that note, thank you very much.

01:27:18.320 | [APPLAUSE]

01:27:20.440 | [APPLAUSE]

01:27:22.440 | [APPLAUSE]

01:27:24.440 | [APPLAUSE]

01:27:26.440 | [APPLAUSE]

01:27:28.440 | [APPLAUSE]

01:27:30.440 | [APPLAUSE]

01:27:32.440 | [APPLAUSE]

01:27:34.440 | [APPLAUSE]

01:27:36.440 | [BLANK_AUDIO]

Deep Learning State of the Art (2020)

Chapters