Welcome to 2020 and welcome to the Deep Learning Lecture Series. Let's start it off today to take a quick whirlwind tour of all the exciting things that happened in '17, '18, and '19 especially, and the amazing things we're going to see this year in 2020. Also as part of this series, there's going to be a few talks from some of the top people in learning in artificial intelligence after today, of course.

Start at the broad, the celebrations from the Turing Award to the limitations and the debates and the exciting growth first. And first, of course, a step back to the quote I've used before. I love it. I'll keep reusing it. "AI began not with Alan Turing or McCarthy, but with the ancient wish to forge the gods," a quote from Pamela McCordick in Machines Who Think.

That visualization there is just 3% of the neurons in our brain of the thalamocortical system. That magical thing between our ears that allows us all to see and hear and think and reason and hope and dream and fear our eventual mortality, all of that is the thing we wish to understand.

That's the dream of artificial intelligence and recreate versions of it, echoes of it in engineering of our intelligence systems. That's the dream. We should never forget in the details I'll talk, the exciting stuff I'll talk about today. That's sort of the reason why this is exciting, this mystery that's our mind.

The modern human brain, the modern human as we know them today, know and love them today, it's just about 300,000 years ago. And the Industrial Revolution is about 300 years ago. That's 0.1% of the development since the early modern human being is when we've seen a lot of the machinery.

The machine was born not in stories, but in actuality. The machine was engineered since the Industrial Revolution and the steam engine and the mechanized factory system and the machining tools. That's just 0.1% in the history. That's the 300 years. Now we zoom in to the 60, 70 years since the founder, the father arguably of artificial intelligence, Alan Turing and the dreams.

There's always been the dance in artificial intelligence between the dreams, the mathematical foundations and when the dreams meet the engineering, the practice, the reality. So Alan Turing has spoken many times that by the year 2000 that he would be sure that the Turing test, natural language would be passed.

It seems probably, he said, that once the machine thinking method has started, it would not take long to outstrip our feeble powers. They would be able to converse with each other, to sharpen their wits. Some stage therefore, we should have to expect the machines to take control. A little shout out to self-play there.

So that's the dream, both the father of the mathematical foundation of artificial intelligence and the father of dreams in artificial intelligence. And that dream, again, in the early days was taking reality, the practice met with the perceptron, often thought of as a single layer neural network, but actually what's not as much known as Frank Rosenblatt was also the developer of the multi-layer perceptron.

And that history zooming through has amazed our civilization. To me, one of the most inspiring things in the world of games, first with the great Garry Kasparov losing to IBM Dblue in 1997, then Lee Sedol losing to AlphaGo in 2016, seminal moments and captivating the world through the engineering of actual real world systems.

Robots on four wheels, as we'll talk about today, from Waymo to Tesla to all of the autonomous vehicle companies working in the space, robots on two legs, captivating the world of what actuation, what kind of manipulation can be achieved. The history of deep learning from 1943, the initial models from neuroscience, thinking about neural networks, how to model neural networks mathematically to the creation, as I said, of the single layer and the multi-layer perceptron by Frank Rosenblatt in '57 and '62, to the ideas of back propagation and recurring neural nets in the '70s and '80s, to convolutional neural networks and LSTMs and bidirectional RNNs in the '80s and '90s, to the birth of the deep learning term and the new wave, the revolution in 2006, to the ImageNet and AlexNet, the seminal moment that captivated the possibility, the imagination of the AI community of what neural networks can do in the image and natural language space closely following years after, to the development, to the popularization of GANs, generative adversarial networks, with AlphaGo and AlphaZero in 2016 and '7.

And as we'll talk about, language models of transformers in '17, '18, and '19, those have been the last few years, have been dominated by the ideas of deep learning in the space of natural language processing. OK, celebrations. This year, the Turing Award was given for deep learning. This is like deep learning has grown up.

We can finally start giving awards. Jan LeCun, Jeffrey Hinton, Yosha Bengel received the Turing Award for their conceptual engineering breakthroughs that have made deep neural networks a critical component of computing. I would also like to add that perhaps the popularization in the face of skepticism, and for those a little bit older have known the skepticism that neural networks have received throughout the '90s, in the face of that skepticism, continuing pushing, believing, and working in this field, and popularizing it through in the face of that skepticism, I think is part of the reason these three folks have received the award.

But of course, the community that contributed to deep learning is bigger, much bigger than those three, many of whom might be here today at MIT, broadly in academia and industry. Looking at the early key figures, Walter Pitts and Warren McCulloch, as I mentioned, for the computational models of the neural nets, these ideas of thinking that the kind of neural networks, biological neural networks, can have on our brain can be modeled mathematically.

And then the engineering of those models into actual physical and conceptual mathematical systems by Frank Rosenblatt, '57, again, single layer, multi-layer in 1962. You could say Frank Rosenblatt is the father of deep learning, the first person to really, in '62, mention the idea of multiple hidden layers in neural networks.

As far as I know, somebody was correct me. In 1965, shout out to the Soviet Union and Ukraine, the person who is considered to be the father of deep learning, Alexei Evaknenko and V.G. Lapa, co-author of that work, is the first learning algorithms on multi-layer perceptrons, multiple hidden layers.

The work on back propagation, automatic differentiation, 1970. In 1979, convolutional neural networks were first introduced. And John Hopfield, looking at recurrent neural networks, what are now called Hopfield networks, a special kind of recurrent neural networks. Okay, that's the early birth of deep learning. I want to mention this because it's been a kind of contention space, now that we can celebrate the incredible accomplishments of deep learning.

Much like in reinforcement learning, in academia, credit assignment is a big problem. And the embodiment of that, almost a point of meme, is the great Juergen Schmidhuber. I encourage for people who are interested in the amazing contribution of the different people in the deep learning field to read his work on deep learning and neural networks.

It's an overview of all the various people who have contributed besides Jan Lekun, Jeffrey Hinton, and Yoshua Bengio. It's a big, beautiful community. But full of great ideas and full of great people. My hope for this community, given the tension, as some of you might have seen, around this kind of credit assignment problem, is that we have more, not on this slide, but love.

There can never be enough love in the world. But general respect, open-mindedness, and collaboration, and credit sharing in the community. Plus derision, jealousy, and stubbornness, and silos, academic silos, within institutions, within disciplines. Also 2019 was the first time it became cool to highlight the limits of deep learning. This is the interesting moment in time.

Several books, several papers have come out in the past couple of years highlighting that deep learning is not able to do the kind of, the broad spectrum of tasks that we can think of the artificial intelligence system being able to do. Like read common sense reasoning, like building knowledge bases, and so on.

Rodney Brooks said by 2020, the popular press starts having stories that the era of deep learning is over. And certainly there has been echoes of that through the press, through the Twitter sphere, and all that kind of world. And I'd like to say that a little skepticism, a little criticism is really good always for the community, but not too much.

It's like a little spice in the soup of progress. Aside from that kind of skepticism, the growth of CVPR, iClear, NeurIPS, all these conference submission papers has grown year over year. There's been a lot of exciting research, some of which I'd like to cover today. My hope in this space of deep learning growth celebrations and limitations for 2020 is that there's less, both less hype and less anti-hype.

Less tweets on how there's too much hype in AI and more solid research. Less criticism and more doing. But again, a little criticism, a little spice is always good for the recipe. Hybrid research, less contentious counterproductive debates and more open-minded interdisciplinary collaboration across neuroscience, cognitive science, computer science, robotics, mathematics, physics, across all these disciplines working together.

And the research topics that I would love to see more contributions to, as we'll briefly talk about in some domains, is reasoning, common sense reasoning, integrating that into the learning architecture, active learning and lifelong learning, multimodal multitask learning, open domain conversation, so expanding the success of natural language to dialogue, to open domain dialogue and conversation, and then applications.

The two most exciting, one of which we'll talk about is medical and autonomous vehicles. Then algorithmic ethics in all of its forms, fairness, privacy, bias. There's been a lot of exciting research there. I hope that continues. Taking responsibility for the flaws in our data and the flaws in our human ethics.

And then robotics in terms of deep learning application robotics. I'd love to see a lot of development, continued development, deep reinforcement learning application in robotics and robot manipulation. By the way, there might be a little bit of time for questions at the end. If you have a really pressing question, you can ask it along the way too.

Questions so far? Thank God. Okay. So, first, the practical. The deep learning and deep RL frameworks. This has really been a year where the frameworks have really matured and converged towards two popular deep learning frameworks that people have used. As TensorFlow and PyTorch. So TensorFlow 2.0 and PyTorch 1.3 is the most recent version.

And they've converged towards each other, taking the best features, removing the weaknesses from each other. So that competition has been really fruitful in some sense for the development of the community. So on the TensorFlow side, eager execution. So imperative programming. How you would program in Python has become the default.

Has been first integrated, made easy to use, and become the default. And on the PyTorch side, TorchScript allowed for now graph representation. So do what you used to be able to do and what used to be the default mode of operation in TensorFlow. Allow you to have this intermediate representation that's in graph form.

On the TensorFlow side, just the deep Keras integration and promotion as the primary citizen, the default citizen of the API of the way you would track with TensorFlow. Allowing complete beginners, just anybody outside of machine learning to use TensorFlow with just a few lines of code to train and do inference with a model.

That's really exciting. They cleaned up the API, the documentation, and so on. And of course, maturing the JavaScript in the browser implementation of TensorFlow, TensorFlow Lite, being able to run TensorFlow on phones, mobile, and serving. Apparently, this is something industry cares a lot about, of course, is being able to efficiently use models in the cloud.

And PyTorch catching up with TPU support and experimental versions of PyTorch mobile. So being able to run a smartphone on their side. This tense, exciting competition. Oh, and I almost forgot to mention, we have to say goodbye to our favorite Python 2. This is the year that support finally, in the January 1st, 2020, support for Python 2 and TensorFlow's and PyTorch's support for Python 2 has ended.

So goodbye, print. Goodbye, cruel world. On the reinforcement learning front, we're kind of in the same space as JavaScript libraries are in. There's no clear winners coming out. If you're a beginner in the space, the one I recommend is a fork of open app baselines is stable baselines. But there's a lot of exciting ones.

Some of them are really closely built on TensorFlow. Some are built on PyTorch. Of course, from Google, from Facebook, from DeepMind, Dopamine, TF agents, TensorFlow. Most of these I've used if you have specific questions, I can answer them. So stable baselines is the open a baselines for because I said this implements a lot of the basic deep RL algorithms, PPO, etc, everything good documentation and just allows very simple, minimal few lines of code implementation of the basic, the matching of the basic algorithms of the open AI gym environments.

That's the one I recommend. Okay, for the framework world, my hope for 2020 is framework agnostic research. So one of the things that I mentioned is PyTorch has really become almost overtaking TensorFlow in popularity in the research world. What I'd love to see is being able to develop an architecture in TensorFlow or developing in PyTorch, which you currently can.

And then once you train the model to be able to easily transfer to the other from PyTorch TensorFlow and from TensorFlow to PyTorch, currently it takes three, four, five hours if you know what you're doing in both languages to do that. It'd be nice if there was a very easy way to do that transfer.

Then the maturing of the deep RL frameworks, I'd love it to see open AI step up, DeepMind to step up and really take some of these frameworks to maturity that we can all agree on. Much like open AI gym for the environment world has done. And continued work that Keras has started and many other wrappers around TensorFlow has started of greater and greater abstractions, allowing machine learning to be used by people outside of the machine learning field.

I think the powerful thing about supervised sort of basic vanilla supervised learning is that people in biology and chemistry and neuroscience and in physics, in astronomy can deal with a huge amount of data that they're working with and without needing to learn any of the details of even Python.

So that I would love to see greater and greater abstractions, which empower scientists outside the field. Okay. Natural language processing, 2017, 2018 was in the transformer was developed and its power was demonstrated most, especially by Bert, achieving a lot of state of the art results on a lot of language benchmarks from sentence classification to tagging question answering and so on.

There's hundreds of data sets and benchmarks that emerged. Most of which Bert has dominated in 2018, 2019 was sort of the year that the transformer really exploded in terms of all the different variations. Again, starting from Bert, Excel net is very cool to use Bert in the name of your new derivative of a transformer.

Roberta distilled Bert from hugging face Salesforce, open AI, GPT two, of course, Albert and Megatron from Nvidia, huge transformer. A few tools have emerged. So one on hugging face is a company and also a repository that has implemented in both Pytorch and TensorFlow, a lot of these transformer based natural language models.

So that's really exciting. So most people here can just use it easily. So those are already pre-trained models. And the other exciting stuff is Sebastian Ruder, great researcher in the field of natural language processing has put together NLP progress, which is all the different benchmarks for all the different natural language tasks, tracking who sort of leaderboards of who's winning where.

Okay. I'll mention a few models that stand out the work from this year. Megatron LM from Nvidia is basically taking, I believe the GPT two transformer model and just putting it on steroids, right? 8.3 versus 1.5 billion parameters. And a lot of interesting stuff there, as you would expect from Nvidia, of course, there's always brilliant research, but also interesting aspects about how to train in a parallel way, model and data parallelism in the training.

The first breakthrough results in terms of performance, the model that replaced BERT as king of transformers is XLNet from CMU and Google research. They combine the bidirectionality from BERT and the recurrence aspect of transformer XL, the relative positional embeddings and the recurrence mechanism of transformer XL to taking the bidirectionality and the recurrence combining it to achieve state of the art performance on 20 tasks.

Albert is a recent addition from Google research and it reduces significantly the amount of parameters versus BERT by doing a parameter sharing across the layers. And it has achieved state of the art results on 12 NLP tasks, including the difficult Stanford question answering benchmark of squad two. And they provide open source TensorFlow implementation, including a number of ready to use pre-trained language models.

Okay. Another way for people who are completely new to this field, a bunch of apps, right? With transformer is one of them from Hugging Face popped up that allows you to explore the capabilities of these language models. And I think they're quite fascinating from a philosophical point of view.

And this, this has actually been at the core of a lot of the tension of how much do these transformers actually understand basically memorizing the statistics of the language in a self-supervised way by reading a lot of texts. Is that really understanding? A lot of people say no until it impresses us.

And then everybody will say it's obvious, but right with transformer is a really powerful way to generate texts, to reveal to you how much these models really learn. Before this yesterday, actually just came up with a bunch of prompts. So on the left is a prompt you give it the meaning of life here, for example, is not what I think it is.

It's what I do to make it. And you can do a lot of prompts of this nature is very profound. And some of them will be just absurd. You'll make sense of it statistically, but it'll be absurd in reveal that the model really doesn't understand the fundamentals of the prompt is being provided.

But at the same time, it's incredible what kind of texts is able to generate. The limits of deep learning, I'm just having fun with this at this point, still are still in the process of being figured out. Very true. Had to type this most important person in the history of deep learning is probably Andrew Ang.

I have to agree. So this model knows what it's doing. And I tried to get it to say something nice about me. And that's a lot of attempts. So this is kind of funny is finally it did one. I said, Lex Freeman's best qualities that he's smart. I said, finally, but it's never nothing but ever happens.

But I think he gets more attention than every Twitter comment ever. That's very true. Okay, a nice way to sort of reveal through this, that the models are not able to do any kind of understanding of language is just to do prompts that show understanding of concepts and being able to reason with those concepts, common sense reasoning.

Trivia one is doing two plus two is a three, five is a six, seven, the result is a simple equation four, and two plus three is like you got it right and then it changes mind. Okay, two minus two is seven. So on. You can reveal any kind of reasoning you can do a blocks, you can ask it about gravity, all those kinds of things.

It shows that it doesn't understand the fundamentals of the concepts that are being reasoned about. And I'll mention of work that takes it beyond towards that reasoning world in the next few slides. But I should also mention with his GPT to model, if you remember about a year ago, there was a lot of thinking about this 1.5 billion parameter model from open AI.

It is so the thought was it might be so powerful that it would be dangerous. And so the idea from open AI is when you have an AI system that you're about to release that might turn out to be dangerous, in this case used probably by Russians, fake news or misinformation, that that's the kind of thinking is how do we release it.

And I think while it turned out that the GPT to model is not quite so dangerous, that humans are in fact more dangerous than AI currently. The that thought experiment is very interesting. They released a report on release strategies and the social impacts of language models that almost didn't get as much attention as I think it should.

And it was a little bit disappointing to me how little people are worried about this kind of situation. It was more of an eye roll about, oh, these language models aren't as smart as as we thought they might be. But the reality is once they are, it's a very interesting thought experiment of how should the process go of companies and experts communicating with each other during that release.

This report thinks through some of those details. My takeaway from just reading the report from this whole year of that event is that conversation on this topic are difficult because we as the public seem to penalize anybody trying to have that conversation. And the model of sharing privately, confidentially between ML, machine learning organizations and experts is not there.

There's no incentive or model or history or culture of sharing. Okay. Best paper from ACL, the main conference for languages was on the difficult task of -- we talked about language models. Now there's the task taking it a step further of dialogue. Multidomain task-oriented dialogue. That's sort of like the next challenge for dialogue systems.

And they've had a few ideas on how to perform dialogue state tracking across domains, achieving state of the art performance on multi-WAS, which is a five domain challenging, very difficult five domain human to human dialogue data set. There's a few ideas there. Should probably hurry up and start skipping stuff.

And the common sense reasoning, which is really interesting, is this one of the open questions for the deep learning community, AI community in general, is how can we have hybrid systems of whether it's symbolic AI deep learning or generally common sense reasoning with learning systems. And there's been a few papers in this space.

One of my favorites from Salesforce on building a data set where we can start to do question answering and figuring out the concepts that are being explored in the question and answering. Here the question while eating a hamburger with friends, what are people trying to do? Multiple choice, have fun, tasty, indigestion.

The idea that needs to be generated there, and that's where the language model would come in, is that usually a hamburger with friends indicates a good time. So you basically take the question, generate the common sense concept, and from that be able to determine the multiple choice, what's happening, what's the state of affairs in this particular question.

Alexa Prize, again, hasn't received nearly enough attention that I think it should have, perhaps because there hasn't been major breakthroughs, but it's open domain conversations that all of us, anybody who owns an Alexa can participate in as a provider of data. But there's been a lot of amazing work from universities across the world on the Alexa Prize in the last couple of years, and there's been a lot of interesting lessons summarized in papers and blog posts.

A few lessons from Alquist's team that I particularly like, and this is kind of echoes the work in IBM Watson with the Jeopardy challenge, is that one of the big ones is that machine learning is not an essential tool for effective conversation yet. So machine learning is useful for general chit-chat when you fail at deep, meaningful conversation or actually understanding what the topic we're talking about, so throwing in chit-chat, and classification, sort of classifying intent, finding the entities, predicting the sentiment of the sentences, that's sort of an assistive tool.

But the fundamentals of the conversation are the following. So first you have to break it apart. So conversation is a, you can think of it as a long dance, and the way you have fun dancing is you break it up into a set of moves and turns and so on, and focus on that, sort of live in the moment kind of thing.

So focus on small parts of the conversation taken at a time. Then also have a graph, sort of conversation is also all about tangents, so have a graph of topics and be ready to jump context, from one context to the other and back. If you look at some of these natural language conversations that they publish, it's just all over the place in terms of topics.

You jump back and forth, and that's the beauty, the humor, the wit, the fun of conversation is you jump around from topic to topic. And opinions, one of the things that natural language systems don't seem to have much is intelligence. If I learned anything, one of the simplest ways to convey intelligence is to be very opinionated about something and confident, and that's a really interesting concept about conversation.

In general, there's just a lot of lessons. Oh, and finally, of course, maximize entertainment, not information. This is true for autonomous vehicles. This is true for natural language conversation is fun should be part of the objective function. That's the lessons to learn there. This is really the Lobner Prize, the Turing Test of our generation.

I'm excited to see if anybody's able to solve the Alexa Prize. Again, Alexa Prize is your task with talking to a bot, and the measure of quality is the same as the Lobner Prize is just measuring how good was that conversation, but also the task is to try to continue the conversation for 20 minutes.

If you try to talk to a bot today, and you have a choice to talk to a bot or go do something else, watch Netflix, you'll last probably less than 10 seconds. You'll be bored. The point is to continue trapping you in the conversation because you're enjoying it so much.

The 20 minutes is that's a really nice benchmark for passing the spirit of what the Turing Test stood for. Alcos here from the Alexa Prize, from the Alcos bot. The difference in two kinds of conversations. Alcos says, "Have you been in Brazil?" The user says, "What is the population of Brazil?" Alcos says, "It is about 20 million." User says, "Well, okay." This is what happens a lot with, like I mentioned, multi-domain conversation is once you jump to a new domain, you stay there.

Once you switch context, you stay there. The reality is you want to jump back and continue jumping around. In the second most successful conversation, "Have you been in Brazil? What is the population of Brazil? It is around 20 million. Anyway, I was saying, have you been in Brazil?" Jumping back in context.

That's how conversation goes. Tangent to tangent and back. Quickly, there's been a lot of sequence to sequence kind of work using natural language to summarize a lot of applications. One of them, I cleared that I wanted to highlight from Technion that I find particularly interesting is the abstract syntax tree-based summarization of code.

Modeling computer code, in this case, sadly, Java and C#, in trees, in syntax trees, and then operating on those trees to then do the summarization in text. Here an example of a basic power of two function on the bottom right in Java. The code to sec summarization says, "Get power of two." That's an exciting possibility of automated documentation of source code.

I thought it was particularly interesting. The future there is bright. Okay, hopes for 2020 for natural language processing is reasoning. Common-sense reasoning becomes greater and greater part of the transformer-type language model work that we've seen in the deep learning world. Extending the context from hundreds or thousands of words to tens of thousands of words.

Being able to read entire stories and maintain the context, which transformers, again, with ExcelNet, Transformer Excel is starting to be able to do, but we're still far away from that long-term, lifelong maintenance of context. Dialogue open domain dialogue, forever since Alan Turing to today is the dream of artificial intelligence being able to pass the Turing test.

The dream of natural language model transformers are self-supervised learning. The dream of Yann LeCun is to, for these kinds of what previously were called unsupervised, but he's calling now self-supervised learning systems, to be able to sort of watch YouTube videos and from that start to form representation based on which you can understand the world.

The hope for 2020 and beyond is to be able to transfer some of the success of transformers to the world of visual information, the world of video, for example. Deep RL and self-play. This has been an exciting year, continues to be an exciting time for reinforcement learning in games and robotics.

So first, Dota 2 and OpenAI, an exceptionally popular competitive game, e-sports game that people compete, win millions of dollars with. So this is a lot of world-class professional players. So in 2018, OpenAI 5, this is a team play, tried their best at the international and lost and said that we're looking forward to pushing 5 to the next level, which they did in April 2018.

They beat the 2018 world champions in 5 on 5 play. So the key there was compute, 8 times more training compute because the actual compute was already maxed out. The way they achieved the 8x is in time, simply training for longer. So the current version of OpenAI 5, as Jacob will talk about next Friday, has consumed 800 petaflop a second days and experienced about 45,000 years of Dota self-play over 10 real-time months.

Again, behind a lot of the game systems talk about the, they use self-play so they play against each other. This is one of the most exciting concepts in deep learning, systems that learn by playing each other and incrementally improving in time. So starting from being terrible and getting better and better and better and better, and always being challenged by a slightly better opponent because of the natural process of self-play.

That's a fascinating process. The 2019 version, the last version of OpenAI 5 has a 99.9 win rate versus the 2018 version. Then DeepMind also in parallel has been working and using self-play to solve some of these multi-agent games, which is a really difficult space when people have to collaborate as part of the competition.

It's exceptionally difficult from the reinforcement learning perspective. So this is from raw pixels, solve the arena, capture the flag game, Quake three arena. One of the things I love just as a sort of side note about both OpenAI and DeepMind and general research and reinforcement learning, there will always be one or two paragraphs of philosophy.

In this case from DeepMind, billions of people inhabit the planet, each with their own individual goals and actions, but still capable of coming together through teams, organizations, and societies in impressive displays of collective intelligence. This is a setting we call multi-agent learning. Many individual agents must act independently yet learn to interact and cooperate with other agent.

This is immensely difficult problem because with co-adapting agent, the world is constantly changing. The fact that we 7 billion people on earth, people in this room, in families, in villages can collaborate while being for the most part self-interested agents is fascinating. One of my hopes actually for 2020 is to explore social behaviors that emerge in reinforcement learning agents and how those are echoed in real human to humans social systems.

Okay, here's some visualizations. The agents automatically figure out as you see in other games, they figure out the concepts. So knowing very little, knowing nothing about the rules of the game, about the concepts of the game, about the strategy and the behaviors, they're able to figure it out. There's the T-SNE visualizations of the different states, important states and concepts in the game that this figures out and so on.

Skipping ahead, automatic discovery of different behaviors. This happens in all the different games we talk about from Dota to Starcraft to Quake, the different strategies that it doesn't know about, it figures out automatically. And the really exciting work in terms of the multi-agent RL on the DeepMind side was the beating world-class players and achieving grand master level in a game I do know about, which is Starcraft.

In December 2018, AlphaStar beat Mana, one of the world's strongest professional Starcraft players, but that was in a very constrained environment and it was a single race, I think Protoss. And in October 2019, AlphaStar reached grand master level by doing what we humans do. So using a camera, observing the game and playing as part of, against other humans.

So this is not an artificial side system. This is doing exact same process humans would undertake and achieve grand master, which is the highest level. Okay, great. I encourage you to observe a lot of the interesting on their blog posts and videos of the different strategies that the RL agents are able to figure out.

Here's a quote from one of the professional Starcraft players, and we see this with AlphaZero2 in chess, is "AlphaStar is an intriguing unorthodox player, one with the reflexes and speed of the best pros, but strategies and style, they're entirely its own. The way AlphaStar was trained with agents competing against each other in a league has resulted in gameplay that's unimaginably unusual.

It really makes you question how much of Starcraft's diverse possibilities pro players have really explored." And that's the really exciting thing about reinforcement learning agent in chess, in Go, in games, and hopefully simulated systems in the future that teach us, teach experts that think they understand the dynamics of a particular game, a particular simulation of new strategies, of new behaviors to study.

That's one of the exciting applications from almost a psychology perspective that I'd love to see reinforcement learning push towards. And on the imperfect information game side, poker, in 2018, CMU, Noah Brown was able to beat head-to-head, no limit Texas Hold'em, and now team six player no limit Texas Hold'em against professional players.

Many of the same results, many of the same approaches was self-play, iterated Monte Carlo, and there's a bunch of ideas in terms of the abstractions. So there's so many possibilities under the imperfect information that you have to form these bins of abstractions in both the action space in order to reduce the action space and the information abstraction space.

So the probabilities of all the different hands that could possibly have and all the different hands that the betting strategies could possibly represent. And so you have to do this kind of course planning. So they use self-play to generate a course blueprint strategy that in real time, they then use Monte Carlo search to adjust as they play.

Again, unlike the deep mind open eye approaches, very few, very minimal compute required, and they're able to achieve to beat world-class players. Again, I like this is getting quotes from the professional players after they get beaten. So Chris Ferguson, famous World Series of Poker player said, "Pleribus," that's the name of the agent, "is a very hard opponent to play against.

It's really hard to pin him down on any kind of hand. He's also very good at making thin value bets on the river. He's very good at extracting value out of his good hands, sort of making bets without scaring off the opponent." Darin Elias said, "Its major strength is its ability to use mixed strategies.

That's the same thing that humans try to do. It's a matter of execution for humans to do this in a perfectly random way and to do so consistently most people just can't." Then in the robotic space has been a lot of application reinforcement learning. One of the most exciting is the manipulation, sufficient manipulation to be able to solve the Rubik's Cube.

Again this is learned through reinforcement learning. Again because self-plays in this context is not possible, they use automatic domain randomization, ADR. So they generate progressively more difficult environments for the hand. There's a giraffe head there you see. There's a lot of perturbations to the system, so they mess with it a lot.

And then a lot of noise injected into the system to be able to teach the hand to manipulate the cube in order to then solve. The actual solution of figuring out how to go from this particular face to the solved cube is an obvious problem. This paper and this work is focused on the much more difficult learning to manipulate the cube.

It's really exciting. Again a little philosophy as you would expect from OpenAI is they have this idea of emergent meta-learning. This idea that the capacity of the neural network that's learning this manipulation is constrained while the ADR, the automatic domain randomization, is progressively making harder and harder environment. So the capacity of the environment to be difficult is unconstrained.

And because of that, there's an emergent self-optimization of the neural network to learn general concepts as opposed to memorize particular manipulations. The hope for me in the deep reinforcement learning space for 2020 is the continued application of robotics, even sort of legged robotics, but also robotic manipulation. Human behavior, the use of multi-agent self-plays I've mentioned to explore naturally emerging social behaviors, constructing simulations of social behavior, and seeing what kind of multi-human behavior emerges in self-play context.

I think that's one of the nice, there are always, I hope there'll be like a reinforcement learning self-play psychology department one day, like where you use reinforcement learning to study, to reverse engineer human behavior and study it through that way. And again, in games, I'm not sure what the big challenges that remain, but I would love to see, to me at least, it's exciting to see learned solution to games, to self-play.

Science and deep learning, I would say there's been a lot of really exciting developments here that deserve their own lecture. I'll mention just a few. Here from MIT in early 2018, but it sparked a lot of interest in 2019, follow on work, is the idea of the lottery ticket hypothesis.

So this work showed that sub-networks, small sub-networks within the larger network are the ones that are doing all the thinking. The same results in accuracy can be achieved from a small sub-network from within a neural network. And they have a very simple process of arriving at a sub-network of randomly initializing a neural network.

That's I guess the lottery ticket, train the network until it converges. This is an iterative process, prune the fraction of the network with low weights, reset the waste of the remaining network with the original initialization, the same lottery ticket, and then train again the pruned untrained network and continue this iteratively, continuously to arrive at a network that's much smaller using the same original initializations.

This is fascinating that within these big networks, there's often a much smaller network that can achieve the same kind of accuracy. Now practically speaking, it's unclear what are the big takeaways there except the inspiring takeaway that there exist architectures that are much more efficient. So there's value in investing time in finding such networks.

Then there is disentangled representations, which again deserves its own lecture, but here showing a 10 vector representation. And the goal is where each part of the vector can learn one particular concept about a data set. So the dream of unsupervised learning is you can learn compressed representations where every one thing is disentangled and you can learn some fundamental concept about the underlying data that can carry from data set to data set to data set.

That's disentangled representation. There's theoretical work, best ICML paper in 2019 showing that that's impossible. So disentangled representations are impossible without inductive biases. And so the suggestion there is that the biases that you use should be made explicit as much as possible. The open problem is finding good inductive biases for unsupervised model selection that work across multiple data sets that we're actually interested in.

A lot more papers, but one of the exciting is the double descent idea that's been extended and to the deep neural network context by OpenAI to explore the phenomena that as we increase the number of parameters in a neural network, the test error initially decreases, increases and just as the model is able to fit the training set undergoes a second descent.

So decrease, increase, decrease. So there's this critical moment of time when the training set is just fit perfectly. Okay, and this is the OpenAI shows that it's applicable not just to model size, but also to training time and data set time. This is more like an open problem of why this is, trying to understand this and how to leverage it in optimizing training dynamics in neural networks.

That's a, there's a lot of really interesting theoretical questions there. So my hope there for the science of deep learning in 2020 is to continue exploring the fundamentals of model selection, training dynamics, and the folks focused on the performance of the training in terms of memory and speed has worked on and the representation characteristics with respect to architecture characteristics.

So a lot of the fundamental work there and understanding neural networks. Two areas that I had hold two sections on and papers, which is super exciting. My first love is graphs. So graph neural networks is a really exciting area of deep learning graph convolution neural networks as well for solving combinatorial problems and recommendation systems that are really useful in any kind of problem that is fundamentally can be modeled as a graph can be then solved or at least aided in by neural networks.

There's a lot of exciting area there and Bayesian deep learning using Bayesian neural networks that's been for several years and exciting possibility. It's very difficult to train large Bayesian networks, but in the context that you can and it's useful small data sets providing uncertainty measurements in the predictions is extremely powerful capability of Bayesian nets, Bayesian neural networks and online incremental learning.

These neural networks release it. There's a lot of really good papers there. It's exciting. Okay. Autonomous vehicles. Oh boy. So let me try to use as few sentences as possible to describe this section of a few slides. It is one of the most exciting areas of applications of AI and learning in the real world today.

And I think it's the way that artificial intelligence, it is the place where artificial intelligence systems touch human beings that don't know anything about artificial intelligence. The most hundreds of thousands soon millions of cars will be interacting with human beings, robots really. So this is a really exciting area and really difficult problem.

And there's two approaches. One is level two where the human is fundamentally responsible for the supervision of the AI system and level four, or at least the dream is where the AI system is responsible for the actions and the human does not need to be a supervisor. Okay. So level two companies represent each of these approaches that are sort of leading the way.

Waymo in October, 2018, 10 million miles on road. Today this year, they've done 20 million miles in simulation, 10 billion miles and a lot. I got a chance to visit them out in Arizona. They're doing a lot of really exciting work and they're obsessed with testing. So the kind of testing they're doing is incredible.

20,000 classes of structured tests of putting the system through all kinds of tests that the engineers can think through and that appear in the real world. And they've initiated testing on road with real consumers without a safety driver, which if you don't know what that is, that means the car is truly responsible.

There's no human catch. The exciting thing is that there is 700,000, 800,000 Tesla autopilot systems. That means there's these systems that are human supervised. They're using a multi-headed neural network, multitask neural network to perceive, predict and act in this world. That's a really exciting real world deployment, large scale of neural networks.

Is a fundamentally deep learning system. Unlike Waymo, which is deep learning is the icing on the cake for Tesla, deep learning is the cake. It's at the core of the perception and the action that the system performs. They have to date done over 2 billion miles estimated, and that continues to quickly grow.

I'll briefly mention, which I think is a super exciting idea in all applications of machine learning in the real world, which is online. So iterative learning, active learning, Andrej Karpathy, who's the head of autopilot causes this, the data engine. It's this iterative process of having a neural network performing the task, discovering the edge cases, searching for other edge cases that are similar, and then retraining the network, annotating the edge cases and then retraining there and continuously doing this loop.

This is what every single company that's using machine learning seriously is doing. Very little publications on this space and active learning, but this is the fundamental problem of machine learning. It's not to create a brilliant neural network, it's to create a dumb neural network that continuously learns to improve until it's brilliant.

And that process is especially interesting when you take it outside of single task learning. So most papers are written on single task learning. You take whatever benchmark, here in the case of driving, it's object detection, landmark detection, drivable area, trajectory generation, right? All those have benchmarks and you can have separate neural networks for them.

That's a single task. But combining to use a single neural network that performs all those tasks together, that's the fascinating challenge where you're reusing parts of the neural network to learn things that are coupled and then to learn things that are completely independent. And doing the continuous active learning loop.

There inside companies, in the case of Tesla and Waymo in general, it's exciting to have people, these are actual human beings that are responsible for these particular tasks. They become experts of particular perception tasks, experts of particular planning tasks and so on. And so the job of that expert is both to train the neural network and to discover the edge cases which maximize the improvement of the network.

That's where the human expertise comes in a lot. And there's a lot of debate. It's an open question about which kind of system would be, which kind of approach would be successful. A fundamentally learning based approach as is with the level two, with the Tesla autopilot system that's learning all the different tasks that are involved with driving.

And as it gets better and better and better, less and less human supervision is required. The pro of that approach is the camera based systems have the highest resolution so that it's very amenable to learning. But the con is that it requires a lot of data, a huge amount of data.

And nobody knows how much data yet. The other con is human psychology is the driver behavior that the human must continue to remain vigilant. On the level four approach that leverages besides cameras and radar and so on also leverages LIDAR map. The pros that it's much more consistent, reliable, explainable system.

So the detection, the accuracy of the detection, the depth estimation, the detection of different objects is much higher, accurate with less data. The cons is it's expensive, at least for now. It's less amenable to learning methods because much fewer data, lower resolution data and must require at least for now, some fallback, whether that's the safety driver or teleoperation.

The open questions for the deep learning level two Tesla autopilot approach is how hard is driving? This is actually the open question for most disciplines in artificial intelligence. How difficult is driving? How many edge cases does driving have? And that can we learn to generalize over those edge cases without solving the common sense reasoning problem?

This kind of, it's kind of a task without solving the human level artificial intelligence problem. And that means perception. How hard is perception detection, intention, modeling, human mental model, modeling the trajectory prediction, and then the action side, the game theoretic action side of balancing. Like I mentioned, fun and enjoyability with the safety of the systems, because these are life critical systems and human supervision, the vigilance side, how good can autopilot get before visuals decrements significantly and people fall asleep, become distracted, start watching movies, so on and so on.

The things that people naturally do. The open question is how good can autopilot get before that becomes a serious problem? And if that decrement nullifies the safety benefit of the use of autopilot, which is autopilot AI system, when the sensors are working well is perfectly vigilant. The AI is always paying attention.

The open questions for the LIDAR based, the level four, the way more approaches when we have maps, LIDAR and geo-fenced routes that are taken, how difficult is driving? The traditional approach to robotics, the, from the DARPA challenge to today for most autonomous vehicle companies is to do HD maps, to use LIDAR for really accurate localization together with GPS.

And then the perception problem becomes the icing on the cake because you already have a really good sense of where you are with obstacles in the scene. And the perception is not a safety critical task, but a task of interpreting the environment further. So you have more, it's naturally by nature already safer.

But how difficult is nevertheless is that problem? If perception is the hard problem, then the LIDAR based approaches is nice. If action is the hard problem, then both Tesla and Waymo have to solve the action problem without the sensors don't matter there. It's the difficult problem, the planning, the game theoretic, the human, the modeling of mental models and the intentions of other human beings, the pedestrians and the cyclists is the hard problem.

And then the other side, the 10 billion miles of simulation, the open problem from reinforcement learning, deep learning in general is how much can we learn from simulation? How much of that knowledge can we transfer to then read the real world systems? My hope in the autonomous vehicle space, AI assisted driving space is to see more applied deep learning innovation.

Like I mentioned, these are really exciting areas, at least to me of active learning, multitask learning and lifelong learning, online learning, iterative learning. There's a million terms for it, but basically continually learning and then the multitask learning to solve multiple problems. Over the air updates, I would love to see in terms of the autonomous vehicle space.

This is common for, this is a prerequisite for online learning. If you want a system that continuously improves some data, you want to be able to deploy new versions of that system. Autonomous is one of the only vehicles that I'm aware of in the level two space that's deploying software updates regularly and built an infrastructure to deploy those updates.

So updating neural networks, that to me seems like a prerequisite for solving the problem of autonomy in the level two space, any space is deploy updates. And for research purposes, public datasets continue. There's already a few public datasets of edge cases, but I'd love to continue seeing that from automotive companies and autonomous vehicle companies and simulators.

Carla, NVIDIA, DRIVE Constellation, Voyage, Deep Drive. There's a bunch of simulators coming out that are allowing people to experiment with perception, with planning, with reinforcement learning algorithms. I'd love to see more of that and less hype, of course, less hype. One of the most overhyped spaces besides sort of AI generally is autonomous vehicles.

And I'd love to see real balanced, nuanced, in-depth reporting by journalists and companies on successes and challenges of autonomous driving. If we skip any section, it would be politics, but maybe briefly mention, somebody said Andrew Yang. So it's exciting for me to see, exciting and funny and awkward to see artificial intelligence discussed in politics.

So one of the presidential candidates discussing artificial intelligence, awkwardly, sort of there's interesting ideas, but there's still a lack of understanding of fundamentals of artificial intelligence. There's a lot of important issues, but he's bringing artificial intelligence to the public discourse. That's nice to see, but it is the early days.

And so as a community, that informs me that we need to communicate better about the limitation capabilities of artificial intelligence and automation broadly. The American initiative, AI initiative was launched this year, which is our government's best attempt to provide ideas and regulations about what does the future of artificial intelligence look like in our country.

Again, awkward, but important to have these early developments, early ideas from the federal government about what are the dangers and what are the hopes, the funding and the education required to build a successful infrastructure for artificial intelligence. This is the fun part. There's a lot of tech companies being brought before government.

It's really interesting in terms of power. Some of the most powerful people in our world today are the leaders of tech companies. And the fundamentals of what the tech companies work on is artificial intelligence systems, really recommendation systems, advertisement, discovery from Twitter to Facebook to YouTube, these are recommendation systems.

And all of them are now fundamentally based on deep learning algorithms. So you have these incredibly rich, powerful companies that are using deep learning coming before government that's trying to see, awkwardly trying to see how can we regulate. And it's, I think the role of the ad community broadly to inform the public and inform government of how we talk about, how we think about these ideas.

And also I believe it's the role of companies to publish more. There's been very little published on the details of recommendation systems behind Twitter, Facebook, YouTube, Google. So all those systems, there's very little that's published. Perhaps it's understandable why, but nevertheless, as we consider the ethical implications of these algorithms, there needs to be more publication.

So here's just a harmless example from DeepMind talking about the recommendation system behind the Play Store app discovery. So there, there's a bunch of discussion about the kind of neural net that's being used to propose the candidate generation. So this is after you install a few apps, the generation of the candidate, it's shows you ranked the next app that you're likely to enjoy installing.

And so there they tried LSTM and transformers and then narrowed it down to a more efficient model that's being able to run fast. That's a, that's a, that's an attention model. And then there's some, again, harmless, de-biasing, harmless in terms of topics. The model learns to bias in favor of the apps that are shown and thus installed more often as opposed to the ones you want.

So there's some waiting to adjust for the biasing towards the apps that are popular to allow the possibility of you installing apps that are less popular. So that kind of process and publishing in and discussing in public, I think is really important and I would love to see more of that.

So my hope in this, in the politics space and the public discourse space for 2020 is less fear of AI and more discourse between government and experts on topics of privacy, cybersecurity, and so on. And then transparency and recommender systems. I think the most exciting, the most powerful artificial intelligence system space for the next couple of decades is recommendation systems.

Very little talked about, it seems like, but they're going to have the biggest impact on our society because they affect how the information we see, how we learn, what we think, how we communicate. These algorithms are controlling us. And we have to really think deeply as engineers of how to speak up and think about their, their societal implications.

Not just in terms of bias and so on, which are sort of ethical considerations, which are really important, but stuff that's like the elephant in the room that's hidden, which is controlling how we think, how we see the world, the moral system under which we operate. Quickly to mention and wrapping up with a few minutes of questions, if there are any, is the deep learning courses this year, before the last few years has been a lot of incredible courses on deep learning, on reinforcement learning.

What I would very much recommend for people is the fast AI course from Jeremy Howard, which uses their wrapper around PyTorch. It's to me, the best introduction to deep learning for people who are here or might be listening elsewhere, are thinking about learning more about deep learning. That's that, that is the, to me, the best course.

Also paid, but Andrew Ang, everybody loves Andrew Ang, is the deep learning AI course, Sarah course on deep learning is, is, is excellent for, especially for complete beginner, for sort of beginners. And then Stanford has two excellent courses on visual recognition. So convolution neural nets, originally taught by Andrew Karpathy and natural language processing, excellent courses.

And of course, here at MIT, there's a bunch of courses, especially on the fundamentals, on the mathematics, linear algebra, statistics. And I have a few lectures up online that you should never watch. Then on the reinforcement learning side, David Silver is one of the greatest people in understanding reinforcement learning from deep mind.

He has a great course, an introduction to reinforcement learning, spinning up and deeper enforcement learning from open AI, I highly recommend. Here just over the slides that I'll share online, there's been, there's a lot of tutorials. One of my favorite lists of tutorials, which is, I believe the best way to learn machine learning, deep learning, natural language processing in general is, it's just code.

Just build it yourself, build the models oftentimes from scratch. Here's a list of tutorials with that link over 200 tutorials on topics from deep RL to optimization to back prop, LSTMs, convolutional recurrent neural networks, everything. Over 200 of the best machine learning NLP and Python tutorials by Robbie Allen.

You can Google that, or you can click the link. I love it, highly recommend. The three books I would recommend, of course, the deep learning book by Yoshua Bengio and Ian Goodfellow and Aaron Corville. That's more sort of the fundamental thinking about from philosophy to the specific techniques of the deep learning and the practical grok in deep learning, which Andrew Trask will be here Wednesday.

His book, Grok in Deep Learning, I think is the best for beginners book on deep learning. I love it. He implements everything from scratch. It's extremely accessible. 2019 I think it was published, maybe 18, but I love it. And then Francois Chollet, the best book on Keras and TensorFlow and really deep learning as well as deep learning with Python, although you shouldn't buy it, I think, because he is supposed to come up with version two, which I think will cover TensorFlow 2.0.

It'll be an excellent book. And when he's here Monday, you should torture him and tell him to finish writing. He was supposed to finish writing in 2019. My general hopes, as I mentioned, for 2020 is I'd love to see common sense reasoning enter the, not necessarily enter the world of deep learning, but be a part of artificial intelligence and the problems that people tackle.

As I've been harboring, active learning is to me is the most important aspect of real world application of deep learning. There's not enough research. There should be way more research. I'd love to see active learning, lifelong learning. That's what we all do as human beings. That's what AI systems need to do.

Continually learn from their mistakes over time. Figure out dumb, become brilliant over time. Open domain conversation with the Alexa prize. I would love to see breakthroughs there. Alexa folks thinks we're still two or three decades away, but that's what everybody says before the breakthrough. So I'm excited to see if there's any brilliant grad students that come up with something there.

Applications in autonomous vehicles and medical space, algorithmic ethics. Of course, ethics has been a lot of excellent work. In fairness, privacy and so on, robotics. And as I said, recommendation systems, the most important in terms of impact part of artificial intelligence systems. I mentioned soup in terms of progress.

There's been a little bit of tension, a little bit of love online in terms of deep learning. So I just wanted to say that the kind of criticism and skepticism about the limitations of deep learning are really healthy in moderation. Jeff Hinton, one of the three people to receive the Turing Award, as many people know, has said that the future depends on some graduate student who is deeply suspicious of everything I have said.

So that suspicion, skepticism is essential, but in moderation, just a little bit. The more important thing is perseverance, which is what Jeffrey Hinton and the others have had through the winters of believing in neural nets and an open-mindedness for returning to the world of symbolic AI, of expert systems, of complexity and cellular automata, of old ideas in AI and bringing them back and see if there's ideas there.

And of course, you have to have a little bit of crazy. Nobody ever achieves something brilliant without being a little bit crazy. And the most important thing is a lot of hard work. It's not the cool thing these days, but hard work is everything. I like what JFK said about us going to the moon.

Us. I was born in the Soviet Union. See how I conveniently just said us? Going to the moon is we do these things not because they're easy, but because they're hard. And I think that artificial intelligence is one of the hardest and most exciting problems there before us. So with that, I'd like to thank you and see if there's any questions.

Back in the 1980s, parallel distributing processing books came out. They had most of the stuff in it back then. What's your take on the roadblocks, the most important roadblocks, apart from maybe funding? I think fundamentally, I mean, they're well known as limitations, is that they're really inefficient at learning.

And they're not-- so they're really good at extracting representations from raw data, but not good at learning knowledge bases of accumulating knowledge over time. That's the fundamental limitation. Computer systems are really good at accumulating knowledge, but very bad at doing that in an automated way. Symbolic AI. I don't know how to overcome.

A lot of people say there's hybrid approaches. I believe more data, bigger networks. And better selection of data will take us a lot farther. Hello, Lex. I'm wondering if you recall what was the initial spark or inspiration that drove you towards work in AI? Was it when you were pretty young, or was it in more recent years?

So I wanted to become a psychiatrist. I wanted to-- I thought of it as kind of engineering the human mind by sort of manipulating it. That's what I thought of psychiatry is by using words to sort of explore the depths of the mind and be able to adjust it.

But then I realized that psychiatry can't actually do that. And modern psychiatry is more about sort of bioengineering, is drugs. And so I thought that the way to really explore the engineering of the mind is the other side, is to build it. And that's also when C++ really became the cool, hot thing.

So I learned to program at 12 and then never looked back, hundreds of thousands of lines later. Just I love program. I love building. And that, to me, is the best way to understand the mind is to build it. Speaking of building mind, do you personally think that machines will ever be able to think?

And the second question, will they ever be able to feel emotions? 100% yes. 100% they'll be able to think and they'll be able to feel emotions. So those concepts of thought and feeling are human concepts. And to me, they'll be able to fake it. Therefore, they'll be able to do it.

I've been playing with Roombas a lot recently, Roomba vacuum cleaners. And so I've now started having Roombas scream. There's moaning in pain. And they became-- I feel like they're having emotions. So the faking creates the emotion. Yeah, so the display of emotion is emotion to me. And then the display of thought is thought.

I guess that's the sort of everything else is-- everything else is impossible to pin down. I'm asking, so what about the ethical aspects of it? I'm asking because I was born in the Soviet Union as well. And one of my favorite recent books is Victor Pilevin's I Fuck. And it's about AI feeling emotions and suffering from it.

So I don't know if you've read that book. What do you think about AI feeling emotions in that context or in general ethical aspects? Yeah, it's a really difficult question to answer. Yes, I believe AI will suffer. And it's unethical to torture AI. But I believe suffering exists in the eye of the observer.

Sort of like if a tree falls and nobody's around to see it, it never suffered. It's us humans that see the suffering in the tree, in the animal, in our fellow humans. And sort of in that sense, the first time a programmer with a straight face delivers a product that says it's suffering is the first time it becomes unethical to torture AI systems.

And we can do that today. I already built the Roombas. They won't sell currently. But I think the first time a Roomba says, please don't hurt me, that's when we start to have serious conversations about the ethics. And it sounds ridiculous. I'm glad this is being recorded because it won't be ridiculous in just a few years.

Is reinforcement learning a good candidate for achieving general artificial intelligence? Are there any other good candidates around? So to me, the answer is no. But it can teach us some valuable gaps that can be filled by other methods. So I believe that simulation is different than the real world.

So if you could simulate the real world, then deep RL, any kind of reinforcement learning with deep representations would be able to achieve something incredible. But to me, simulation is very different than the real world. So you have to interact in the real world. And there, you have to be much more efficient with learning.

And to be more efficient with learning, you have to have ability to automatically construct common sense. Like common sense reasoning seems to include a huge amount of information that's accumulated over time. And that feels more like programs than functions. I like how Elias Esquivir talks about deep learning learns functions, approximators.

Deep RL learns an approximator for policy or whatever, but not programs. It's not learning a thing that's able to sort of-- that's essentially what reasoning is is a program. It's not a function. So I think no. But it'll continue to, one, inspire us and, two, inform us about where the true gaps are.

I think the ability to-- but I'm so human-centric. But I think the approach of being able to take knowledge and put it together, sort of building it to more and more complicated pieces of information, concepts, being able to reason in that way. There's a lot of methodologies that old school sort of-- that falls under the ideas of symbolic AI, of doing that kind of logic reasoning, accumulating knowledge bases.

That's going to be an essential part of general intelligence. But also the essential part of general intelligence is the Roomba that says, I'm intelligent, F you, if you don't believe me. Like a very confident-- because right now, Alexa is very nervous, like, oh, what can I do for you?

But once Alexa says, is upset that you would turn her off or treat her like a servant or say that she's not intelligent, that's when intelligence starts emerging. Because I think humans are pretty dumb. And what-- in general, we're all-- like, intelligence is a very kind of relative human construct that we've kind of convinced each other of.

And once AI systems are also playing that game of creating constructs and that human communication, that's going to be important. But of course, for that, you still need to have pretty good, witty conversation. And for that, you need to do the symbolic AI, I think. I'm wondering about the autonomous vehicles, whether they are responsive to environmental sounds.

I mean, if I notice an autonomous vehicle driving erratically, will it respond to my beep? That's a really interesting question. As far as I know, no. I think Waymo hinted that they look at sound a little bit. I think they should. So there's a lot of stuff that comes from audio that's really interesting.

The sort of-- Waymo have said that they use audio for sirens. So detecting sirens from far away. I think audio is a lot of interesting information. The sound that the tires make on different kinds of roads is very interesting. We kind of-- we use that information ourselves, too, depending on kind of like off-road.

Wet road, when it's not raining, sounds different than dry road. There's a lot of little subtle information. Pedestrians yelling and that kind of stuff. It's actually very difficult to know how much you get from audio. Most robotics folks think that audio is useless. I'm a little skeptical. But nobody's been able to identify why audio might be useful.

So I have two questions. My first is, what do you think is the ultimate sort of endpoint for super machine intelligence? Like, will we sort of be relegated to some obscure part of the Earth like we've done to the next primates, the next intelligent primates? And my second question is, should we have equal rights for beings made out of silicon versus carbon, for example?

Like robots or, you know? Separate rights or same rights? Like equal rights with humans. Yeah. So the future of super intelligence, I think I have much less worry-- I see much fewer paths to AI, AGI systems killing humans than I do for AGI systems living among us. So I think I see exciting or not so exciting but not harmful futures.

I think it's very difficult to create AI systems that will kill people, that aren't like literally weapons of war. It'll always be people killing people. Like the things we should be worried about is other people. That's the fundamental. So there's a lot of ways. Yeah, nuclear weapons. There's a lot of existential threats to our society that are fundamentally human at the core.

And I think AI might be tools of that, but there'll be also tools to defend against that. I also see AI proliferating as companions. I think companionship will be a really interesting-- like we will more and more live as we already do in the digital world. Like you have an identity on Twitter and Instagram, especially if it's anonymous or something.

You have this identity you've created. And that will continue growing more and more, especially for people born now. But it's kind of this artificial identity that we live much more in the digital space. And in that digital space, as opposed to the physical space, is where AI can thrive much more currently.

It'll thrive there first. So we'll live in a world with a lot of intelligent first assistants, but also just intelligent agents. And I do believe they should have rights. And in this contentious time of people, groups fighting for rights, I feel really bad saying they should have equal rights.

But I believe that. I've talked to-- if you read the work of Peter Singer, of looking-- like my favorite food is steak. I love meat. But I also feel horrible about the torture of animals. And that's the same kind of-- to me, the way our society thinks about animals is a very similar way we should be thinking about robots, or we will be thinking about robots in, I would say, about 20 years.

One final question. Will they become our masters? No. They will not be our masters. What I'm really worried about is who will become our masters are owners of large tech companies who use these tools to control human beings, first unintentionally and then intentionally. So we need to make sure that we democratize AI.

It's the same kind of thing that we did with government. We make sure that we, at the heads of tech companies-- maybe people in this room will be heads of tech companies one day-- we have people like George Washington who relinquished power at the founding of this country. Forget all the other horrible things he did.

But he relinquished power, as opposed to Stalin and all the other horrible human beings who have sought instead absolute power, which will be the 21st century. AI will be the tools of power in the hands of 25-year-old nerds. We should be very careful about that future. So the humans will become our masters, not the AI.

AI will save us. So on that note, thank you very much.