Self-Driving Cars: State of the Art (2019)

00:00:00.000 | Today I'd like to talk about

00:00:01.640 | the state of the art of autonomous vehicles,

00:00:03.960 | how I see the landscape,

00:00:05.800 | how others see the landscape,

00:00:07.780 | what we're all excited about,

00:00:09.360 | ways to solve the problem,

00:00:11.320 | and what to look forward to in 2019

00:00:14.720 | as we also get to hear from the different perspectives

00:00:18.200 | from the various leaders in industry

00:00:19.820 | and autonomous vehicles in the next few,

00:00:22.000 | next couple of weeks and next few days.

00:00:23.960 | So the problem, the mission, the dream,

00:00:28.240 | the thing that we're trying to solve,

00:00:30.520 | for many it may be about entrepreneurial possibilities

00:00:33.600 | of making money and so on,

00:00:34.800 | but really it's about improving access to mobility,

00:00:39.600 | moving people around the world that don't have that ability,

00:00:42.880 | whether it has to do with age

00:00:44.880 | or purely access of where you live.

00:00:47.360 | We want to increase the efficiency

00:00:51.120 | of how people move about,

00:00:52.840 | the ability to be productive

00:00:54.840 | in the time we spend in traffic and transportation.

00:00:58.880 | One of the most hated things in terms of stress and motion,

00:01:03.800 | the thing in our lives that if we could just

00:01:06.680 | with a snap of a finger remove is traffic.

00:01:09.180 | So the ability to convert that into efficiency,

00:01:13.160 | into a productive aspect,

00:01:14.360 | into a positive aspect of life.

00:01:16.200 | And really the most important thing,

00:01:18.960 | at least for me and for many of us working in this space,

00:01:21.520 | is to save lives,

00:01:23.200 | prevent crashes that lead to injuries,

00:01:25.480 | prevent crashes that lead to fatalities.

00:01:27.520 | Here's a counter, every 23 seconds,

00:01:29.880 | somebody in the world dies in a car auto crash.

00:01:33.580 | It should be a sobering, it is for me,

00:01:36.560 | thing that I think about every single day.

00:01:39.040 | You go to bed, you wake up,

00:01:40.200 | you work on all the deep learning methods,

00:01:41.680 | all the different papers we're publishing,

00:01:43.720 | everything we're trying to push forward

00:01:45.200 | is really to save lives at the beginning

00:01:48.760 | and at the end, that is the main goal.

00:01:52.240 | So with that groundwork, with that idea,

00:01:54.220 | with that base of mission that we're all working towards

00:01:57.400 | from the different ideas and different perspectives,

00:02:00.560 | I would like to review what happened in 2018.

00:02:03.040 | So first, Waymo has done incredible work

00:02:09.800 | in deploying and testing their vehicles in various domains

00:02:13.880 | and have in October reached the mark of 10 million miles

00:02:18.480 | driven autonomously, which is an incredible accomplishment.

00:02:21.240 | It's truly a big step for fully autonomous vehicles

00:02:26.240 | in terms of deployment and obviously is growing

00:02:30.580 | and growing by day and we'll have Drago here from Waymo

00:02:35.120 | to talk about their work there.

00:02:36.900 | Then on the L2, on the semi-autonomous side,

00:02:40.240 | that's the pair, that's the mirror side of this equation.

00:02:44.320 | The other incredible number

00:02:46.840 | that's perhaps less talked about

00:02:49.040 | is the one billion mile mark reached by Tesla

00:02:53.720 | in the semi-autonomous driving of Autopilot.

00:02:56.680 | Autopilot is a system that's able to control

00:02:58.840 | its position in the lane, center itself in the lane,

00:03:01.440 | it's able to control the longitudinal movement,

00:03:03.780 | so not follow a vehicle

00:03:05.800 | when there's a vehicle in front and so on.

00:03:07.780 | But the degree of its ability to do so

00:03:11.120 | is the critical thing here,

00:03:12.500 | is the ability to do so for many minutes at a time,

00:03:15.240 | even hours at a time, especially on highway driving,

00:03:18.000 | that's the critical thing.

00:03:19.120 | And the fact that they've reached one billion

00:03:21.480 | with a B miles is an incredible accomplishment.

00:03:24.940 | All of that from the machine learning perspective is data,

00:03:28.320 | that's data and all of the Autopilot miles

00:03:32.440 | are driven with the primary sensor being a camera,

00:03:37.440 | so it's computer vision.

00:03:39.480 | And how does computer vision work in modern day?

00:03:42.080 | Especially with the second iteration of Autopilot hardware,

00:03:46.240 | there's a neural network,

00:03:47.200 | there's a set of neural networks behind it.

00:03:49.100 | That's super exciting.

00:03:50.400 | That is probably the largest deployment of neural networks

00:03:55.400 | in the world that has a direct impact on a human life

00:04:02.340 | that's able to decide,

00:04:07.420 | that's able to make life critical decisions

00:04:10.960 | many times a second over and over.

00:04:13.200 | That's incredible.

00:04:14.240 | You go from the step of image classification

00:04:17.120 | on ImageNet and you sit there with a TensorFlow

00:04:21.040 | and you're very happy there,

00:04:22.200 | you're able to achieve a 99.3 accuracy

00:04:25.400 | with a state of the art algorithm.

00:04:26.960 | You take from that a step towards there's a human life,

00:04:31.560 | your parents driving, your grandparents driving this,

00:04:36.560 | your children driving the system

00:04:39.400 | and there's a neural network making the decision

00:04:41.040 | of whether they live.

00:04:43.120 | So that one billion mark is an incredible accomplishment.

00:04:47.040 | And on the sobering side,

00:04:50.120 | from various perspectives, the fatalities.

00:04:54.800 | There's been two fatalities that happened in March of 2018.

00:04:58.600 | One in the fully autonomous side of things

00:05:00.720 | with Uber in Tempe, Arizona hitting a pedestrian

00:05:04.880 | and leading to a pedestrian fatality.

00:05:08.320 | And on the semi-autonomous side of Tesla Autopilot,

00:05:12.400 | the third fatality that Tesla Autopilot led to

00:05:15.680 | and the one in 2018 is in Mountain View, California

00:05:20.560 | when Tesla slammed into a divider, killing his driver.

00:05:25.560 | Now, the two aspects here that are sobering

00:05:32.320 | and really important to think about

00:05:33.600 | as we talk about the progression of autonomous vehicles,

00:05:38.000 | proliferation in our world is our response as a public,

00:05:42.240 | as from the general public to the engineers,

00:05:45.640 | to the media and so on,

00:05:46.920 | how we think about these fatalities.

00:05:48.640 | And obviously there's a disproportionate amount

00:05:51.520 | of attention given to these fatalities.

00:05:53.440 | And that's something as engineers

00:05:54.960 | you have to also think about,

00:05:56.400 | that the bar is much higher on every level

00:05:59.280 | in terms of performance.

00:06:01.280 | So in order to success, as I'll argue,

00:06:03.760 | in order to design successful autonomous vehicles,

00:06:06.400 | those vehicles will have to take risks.

00:06:09.400 | And when the risks don't pan out,

00:06:13.360 | the public, if the public doesn't understand

00:06:17.200 | the general problem that we're tackling,

00:06:20.360 | the goal, the mission, that those risks,

00:06:22.800 | when they don't, the risks that are taken

00:06:25.800 | can have significant detrimental effect to the progress

00:06:30.400 | in this autonomous vehicle space.

00:06:32.320 | So that's something we really have to think about.

00:06:34.480 | That's our role as engineers and so on.

00:06:36.760 | Question, yeah.

00:06:37.720 | So the question was, do we know the rate of fatalities

00:06:41.800 | per mile of vehicle driven,

00:06:43.880 | which is at the crudest level

00:06:46.040 | how people think about safety.

00:06:47.800 | So there's about 80, 90, 100 million miles driven

00:06:51.560 | in manually controlled cars at every fatality.

00:06:55.280 | So one fatality per, depending on which numbers you look at,

00:06:59.240 | it's 80 to 100 million miles.

00:07:00.920 | And the Tesla vehicle, for example,

00:07:04.880 | the fatality is, well, we could just take the one billion

00:07:08.440 | and divide it by three.

00:07:11.220 | Now it's apples and oranges in comparison.

00:07:14.160 | And that's something actually that we're working on

00:07:16.320 | to make sure that we compare it correctly,

00:07:19.040 | compare the aspects of manual miles

00:07:22.240 | that directly are comparable to the autopilot miles.

00:07:26.480 | So autopilot is a modern vehicle that's much safer.

00:07:29.380 | Tesla is a modern vehicle that's much safer

00:07:31.080 | than the general population of manually driven vehicles.

00:07:34.440 | Autopilot is driven on only particular kinds of roads,

00:07:37.320 | on the highway, primarily most of the miles.

00:07:40.720 | The kinds of people that drive autopilot,

00:07:42.720 | all these kinds of factors need to be considered

00:07:44.560 | when you compare the two.

00:07:46.180 | But when you just look at the numbers,

00:07:48.360 | Tesla autopilot is three times safer

00:07:50.600 | than manually driven vehicles.

00:07:52.400 | But that's not the right way to look at it.

00:07:54.780 | And for anyone that's ever taken a statistics class,

00:07:58.720 | three fatalities is not a large number

00:08:03.720 | by which to make any significant conclusions.

00:08:09.760 | Nevertheless, that doesn't stop the media,

00:08:11.800 | the New York Times and everybody

00:08:13.060 | from responding to a single fatality.

00:08:15.120 | Which PR and marketing aspects of these different companies

00:08:20.280 | are very sensitive to.

00:08:21.860 | Which is of course troubling and concerning

00:08:23.640 | for an engineer that wants to save lives.

00:08:26.480 | But it's something that we have to think about.

00:08:28.620 | Okay, 2018 in review continued.

00:08:31.600 | There's been a lot of announcements,

00:08:34.720 | or rather actual launches of public testing

00:08:39.960 | of autonomous taxi services.

00:08:42.040 | So companies that are on public roads

00:08:45.320 | have been delivering real people

00:08:48.000 | from one location to another.

00:08:49.880 | Now there's a lot of caveats.

00:08:51.260 | In many of these cases, it's very small scale,

00:08:54.600 | just a few vehicles.

00:08:56.060 | In most cases, it's very low speed

00:08:58.200 | in a constrained environment, in a constrained community.

00:09:02.000 | And almost always, really always with a safety driver,

00:09:06.480 | there's a few exceptions for demonstration purposes.

00:09:09.240 | But there's always an actual driver in the seat.

00:09:11.640 | Some of the brilliant folks representing these companies

00:09:14.840 | will speak in this course.

00:09:16.200 | Is Voyage doing it in an isolated community?

00:09:20.640 | Awesome work they're doing in villages in Florida.

00:09:22.560 | Optimus Ride here in Boston,

00:09:25.560 | doing in the community in Union Point.

00:09:27.640 | Drive AI in Texas.

00:09:30.660 | Maine Mobility expanding beyond Detroit,

00:09:33.460 | but really most operations in Detroit.

00:09:35.880 | Waymo has launched its service.

00:09:37.520 | Waymo One that's gotten some publicity in Phoenix, Arizona.

00:09:42.440 | Neuro doing zero occupancy deliveries

00:09:47.160 | of groceries autonomously.

00:09:50.100 | So we didn't say it has to be delivering humans,

00:09:52.000 | it's delivering groceries autonomously.

00:09:54.560 | Uber is quietly or not so quietly resumed

00:09:58.840 | its autonomous vehicle taxi service testing in Pittsburgh

00:10:05.320 | in a very careful, constrained way.

00:10:09.080 | Aptiv, after acquiring Carling-Yema,

00:10:14.080 | is a new autonomy, has been doing extensive,

00:10:19.080 | large-scale taxi service testing

00:10:22.800 | everywhere from Vegas to Boston here,

00:10:25.360 | to Pittsburgh and in Singapore, of course.

00:10:29.580 | Aurora that spoke here last time,

00:10:34.920 | the head of Tesla Autopilot that launched Aurora,

00:10:37.560 | and the Chris Hermsen behind this young upstart company

00:10:41.800 | is doing testing in San Francisco and Pittsburgh.

00:10:44.080 | And then Cruise, Kyle will be here to talk from GM,

00:10:47.240 | is doing testing in San Francisco, Arizona, and Michigan.

00:10:50.080 | So when we talk about predictions,

00:10:54.040 | I'll talk about a few people predicting

00:10:56.440 | when we're going to have autonomous vehicles.

00:10:59.360 | And when you yourself think about what it means,

00:11:02.720 | when will they be here?

00:11:04.800 | When will autonomous vehicles arise

00:11:06.600 | such that the Uber that you call will be autonomous

00:11:09.160 | and not with a populated by a driver?

00:11:11.360 | So the thing we have to think about

00:11:14.720 | is what we think about how we define autonomous,

00:11:19.200 | what that experience looks like.

00:11:21.080 | And most importantly, in these discussions,

00:11:23.760 | we have to think about scale.

00:11:25.260 | So we here at MIT, our group,

00:11:29.320 | MIT Human-Centered Autonomous Vehicle,

00:11:31.360 | we have a fully autonomous vehicle

00:11:32.880 | that people can get in if you would like,

00:11:34.880 | and it will give you a ride in a particular location.

00:11:37.480 | But that's one vehicle, it's not a service,

00:11:41.320 | and it only works on particular roads,

00:11:43.640 | it's extremely constrained.

00:11:45.200 | In some ways, it's not much different

00:11:46.920 | than most of the companies that we're talking about today.

00:11:50.080 | Now, scale here, there's a magic number,

00:11:52.780 | I'm not sure what that is,

00:11:53.740 | but for the purpose of this conversation,

00:11:55.880 | it says 10,000, where there's a meaningful deployment.

00:11:59.740 | When it's truly going beyond that prototype demo mode

00:12:04.620 | to where everything's under control,

00:12:06.320 | to where it's really touching the general population

00:12:09.300 | in a fundamental way, scale is everything here.

00:12:12.540 | And it starts, let's say, at 10,000.

00:12:14.980 | Just to give you for reference,

00:12:16.200 | there's 46,000 active Uber drivers in New York City.

00:12:19.460 | So that's what 10,000 feels like.

00:12:21.180 | 25, 30% of the Uber drivers in New York City

00:12:26.260 | all of a sudden become passengers.

00:12:31.260 | So the predictions.

00:12:36.300 | I'm not a marketing PR person,

00:12:38.580 | so I don't understand why everybody

00:12:40.180 | has to make a prediction, but they all seem to.

00:12:43.660 | All the major automakers have made a prediction

00:12:45.700 | of when they'll have a deploy,

00:12:47.320 | when they will be able to deploy autonomous vehicles.

00:12:50.240 | Tesla has made in early 2017,

00:12:56.700 | a prediction that they will have autonomous vehicles 2018.

00:12:59.780 | In 2018, they've now adjusted the prediction to 2019.

00:13:04.260 | Nissan, Honda, Toyota have made prediction for 2020

00:13:09.260 | under certain constraints in highway urban.

00:13:12.140 | Hyundai and Volvo has in 2021,

00:13:16.220 | with BMW and Ford, Ford saying at scale.

00:13:20.300 | So a large scale deployment in 2021.

00:13:23.540 | And Chrysler in '21,

00:13:24.820 | and Daimler saying in the early '20s.

00:13:27.700 | So there is the predictions that are extremely optimistic

00:13:32.700 | that are perhaps driven by the instinct

00:13:38.060 | that the company has to declare

00:13:41.340 | that they're at the cutting edge of innovation.

00:13:44.220 | And then there is many of the leading engineers

00:13:46.700 | behind the leading these teams,

00:13:49.100 | including Carl and Yamaha and Gil Pratt from MIT,

00:13:53.420 | who injects a little bit of caution

00:13:57.940 | and grounded ideas about how difficult it is

00:14:04.460 | to remove the human from the loop of automation.

00:14:07.660 | So Carl says that basically tele-operation

00:14:11.820 | kind of gives this analogy of an elevator.

00:14:14.060 | You know, an elevator is fully autonomous,

00:14:16.100 | but there's still a button to call for help

00:14:18.660 | if something happens.

00:14:20.300 | And that's how he thinks about autonomous vehicles,

00:14:22.780 | even with greater and greater degree of automation,

00:14:25.020 | there's still going to have to be a human in the loop.

00:14:27.340 | There's still going to be a way

00:14:28.620 | to contact a human to get help.

00:14:31.260 | And Gil Pratt and Toyota,

00:14:33.980 | and they're making some announcements at CES,

00:14:36.300 | basically saying that the human in the loop

00:14:37.900 | is the fundamental aspect

00:14:39.100 | that we need to approach this problem.

00:14:40.340 | And removing the human from consideration

00:14:43.140 | is really, really far away.

00:14:45.060 | And Gil, historically and currently,

00:14:49.560 | is one of the sort of the great roboticists in the world

00:14:51.780 | that defined a lot of the DARPA challenges

00:14:53.900 | and a lot of our progress, historically speaking,

00:14:56.860 | up to this point.

00:14:58.300 | So really the full spectrum,

00:14:59.740 | we can think of it as the Elon Rodney spectrum

00:15:04.460 | of optimism versus pessimism.

00:15:07.300 | The Elon Musk, who's extremely bold and optimistic

00:15:11.580 | about his predictions.

00:15:13.460 | I often connect with this kind of thinking

00:15:17.140 | because sometimes you have to believe

00:15:18.360 | the impossible is possible in order to make it happen.

00:15:21.860 | And then there's Rodney,

00:15:23.940 | also one of the great roboticists,

00:15:26.180 | the former head of C-Cell, the AI laboratory here,

00:15:30.580 | is a little bit on the pessimistic side.

00:15:33.000 | So for Elon, a fully autonomous vehicle

00:15:35.180 | will be here in 2019.

00:15:37.160 | For Rodney, the vehicles are really fully autonomous,

00:15:41.500 | or beyond 2050.

00:15:43.140 | But he believes in the 30s,

00:15:46.420 | there will be a significant,

00:15:49.260 | a major city will be able to allocate

00:15:51.740 | a significant region of that city

00:15:56.020 | where manual driving is fully banned,

00:15:58.540 | which is the way he believes those vehicles,

00:16:01.260 | autonomous vehicles can really proliferate

00:16:02.800 | when you ban manually driven vehicles in certain parts.

00:16:06.040 | And in the 40s, 2045 or beyond,

00:16:09.380 | majority of US cities will ban manually driven vehicles.

00:16:13.600 | Of course, the quote from Elon Musk in 2017

00:16:18.180 | is that my guess is that in probably 10 years,

00:16:22.540 | it will be very unusual for cars to be built

00:16:25.580 | that are not fully autonomous.

00:16:27.300 | So we also have to think about the long tail of the fact

00:16:32.300 | that many people drive cars that are 10 years old,

00:16:35.340 | 20 years old.

00:16:36.160 | So even when you have every car is built

00:16:38.700 | that's fully autonomous,

00:16:39.820 | it's still gonna take time

00:16:41.300 | for that dissipation of vehicles to happen.

00:16:44.340 | And so my own view beyond predictions,

00:16:48.320 | to take a little pause into the ridiculous and the fun

00:16:52.400 | to explain the view.

00:16:53.600 | Yes, that is me playing guitar in our autonomous vehicle.

00:16:59.760 | Now the point of this ridiculous video

00:17:02.780 | and embarrassing I should never played it.

00:17:04.880 | Yeah, okay, I think it's gonna be over soon.

00:17:08.960 | Now for those of you born in the 90s, that's classic rock.

00:17:15.000 | So the point I'm trying to make beyond predictions

00:17:18.560 | is that autonomous vehicles will not be adopted

00:17:21.680 | by human beings in the near term,

00:17:24.120 | in the next 10, 15 years, because they're safer.

00:17:27.360 | Safety is not going to, they may be safer,

00:17:30.400 | but they're not going to be so much safer

00:17:33.440 | that that's going to be the reason you adopt.

00:17:36.600 | It's not gonna be because they get you

00:17:38.200 | to the location faster.

00:17:39.480 | Everything we see with autonomy

00:17:41.280 | is they're going to be slower

00:17:42.920 | until majority of the fleet is autonomous.

00:17:46.880 | They're cautious and therefore slower

00:17:49.520 | and therefore more annoying in the way we think about

00:17:51.920 | actually how we navigate this world.

00:17:53.680 | We take risk, we drive assertively with speed

00:17:56.180 | over the speed limit all the time.

00:17:57.760 | That is not how autonomous vehicles today operate.

00:18:00.480 | So they're not gonna get us there faster.

00:18:02.240 | And for every promise, every hope

00:18:05.200 | that they're going to be cheaper,

00:18:06.760 | really there's still significant investment going into them.

00:18:11.160 | And there's not good economics in the near term

00:18:14.280 | of how to make them obviously significantly cheaper.

00:18:17.360 | What I think Uber and Lyft has taken over

00:18:22.360 | the taxi service because of the human experience.

00:18:26.300 | In the same way, autonomy will only take over

00:18:29.080 | if not take over, be adopted by human beings

00:18:32.400 | if it creates a better human experience.

00:18:34.800 | If there's something about the experience

00:18:37.300 | that you enjoy the heck out of.

00:18:40.080 | This video and many others that we're putting out

00:18:44.360 | shows that natural language communication,

00:18:46.840 | the interaction with the car,

00:18:48.540 | the ability of the car to sense everything you're doing

00:18:50.920 | from the activity of the driver to the driver's attention

00:18:55.080 | and being able to transfer control back and forth

00:18:57.680 | in a playful way, but really in a serious way.

00:19:00.980 | Also that's personalized to you.

00:19:05.620 | That's really the human experience,

00:19:08.120 | the efficiency of the human experience,

00:19:09.960 | the richness of the human experience,

00:19:11.520 | that is what we need to also solve.

00:19:14.400 | That's something you have to think about

00:19:15.620 | because many of the people that'll be speaking at this class

00:19:18.840 | and many of the people that are working on this problem

00:19:21.820 | are not focused on the human experience.

00:19:24.760 | It's a kind of afterthought

00:19:27.000 | that once we solve the autonomous vehicle problem,

00:19:29.240 | it'll be fun as hell to be in that car.

00:19:31.480 | I believe you first have to make it fun as hell

00:19:34.880 | to be in the car and then solve

00:19:36.480 | the autonomous vehicle problem jointly.

00:19:40.860 | In the language that we're talking about here,

00:19:43.820 | there's several levels of autonomy that are defined

00:19:46.520 | from level zero to level four, level zero,

00:19:48.420 | no automation, four and five,

00:19:50.900 | level three, four and five, increasing automation.

00:19:53.420 | So level two is when the driver is still responsible.

00:19:56.860 | Level three, four, five is when there's

00:19:58.620 | less and less responsibility, but really in three, four, five

00:20:02.380 | there's parts of the driving

00:20:04.840 | where the liability is on the car.

00:20:07.180 | So there's only really two, as far as I'm concerned,

00:20:09.760 | levels, human-centered autonomy and full autonomy.

00:20:13.220 | Human-centered means the human is responsible.

00:20:15.820 | Full autonomy means the car is responsible,

00:20:19.520 | both on the legal side, the experience side,

00:20:23.460 | and the algorithm side.

00:20:25.900 | That means full autonomy does not allow for teleoperation.

00:20:30.900 | So it doesn't allow for the human to step in

00:20:35.540 | and remotely control the vehicle

00:20:37.000 | because that means the human is still in the loop.

00:20:39.460 | It doesn't allow for the 10-second rule

00:20:41.740 | that it's gonna be fully autonomous,

00:20:44.540 | but once it starts warning you,

00:20:45.820 | you have 10 seconds to take over.

00:20:47.560 | No, it's not fully autonomous.

00:20:49.620 | We cannot guarantee safety in any situation.

00:20:53.940 | It has to be able to,

00:20:54.940 | if the driver doesn't respond in 10 seconds,

00:20:57.220 | it has to be able to find safe harbor.

00:20:58.740 | It has to be able to pull off to the side of the road

00:21:01.020 | without hurting anybody else to find safety.

00:21:04.940 | So that's the fully autonomous challenge.

00:21:08.120 | And so how do we envision these two levels of automation

00:21:12.780 | proliferating society, getting deployed at a mass scale?

00:21:16.120 | The 10,000, 10 million, beyond.

00:21:19.620 | On the fully autonomous side,

00:21:21.160 | the way to think about it with the predictions

00:21:27.200 | that we're talking about here

00:21:29.380 | is there's several different possibilities

00:21:31.240 | of how to deploy these vehicles.

00:21:35.620 | One is last mile delivery of goods and service,

00:21:40.620 | like the groceries.

00:21:42.780 | These are zero occupancy vehicles delivering groceries

00:21:45.240 | or delivering human beings at the last mile.

00:21:48.860 | What the last mile means is

00:21:50.740 | it's slow moving transport to the destination

00:21:54.640 | where most of the tricky driving along the way

00:21:56.900 | is done manually, and then the last mile delivery

00:21:59.060 | in the city, in the urban environment

00:22:01.220 | is done by zero occupancy autonomous vehicles.

00:22:06.220 | Trucking on the highway, possibly with platooning

00:22:09.980 | where a sequence of trucks follow each other.

00:22:12.300 | So in this, what people think about

00:22:14.580 | as a pretty well-defined problem of highway driving

00:22:19.580 | with lanes well-marked, well-mapped,

00:22:23.340 | routes throughout the United States,

00:22:25.420 | and globally on the highway driving is automatable.

00:22:28.900 | The specific urban routes,

00:22:30.560 | kind of like what a lot of these companies are working on,

00:22:33.340 | defining this taxi service

00:22:36.220 | and a personalized public transport.

00:22:38.380 | There's certain pickup locations you're allowed to go to,

00:22:41.580 | there are certain drop-off locations, that's it.

00:22:43.820 | It's kind of like taking the train here,

00:22:45.680 | but as opposed to getting on the train

00:22:47.500 | with 100 other people or bus,

00:22:50.180 | you're getting on a car when you're alone

00:22:52.700 | or with one other person.

00:22:53.940 | The closed communities, something Oliver Cameron

00:22:58.880 | with Voyage is working on defining and Optimus Ride,

00:23:03.060 | defining a particular community

00:23:04.740 | that you now have a monopoly over,

00:23:08.540 | that you define the constraints,

00:23:09.700 | you define the customer base,

00:23:11.000 | and then you just deliver the vehicles,

00:23:12.580 | you map the entire road,

00:23:14.660 | you have slow moving transport that gets people

00:23:16.660 | from A to B, anywhere in that community.

00:23:18.840 | And then there's the world

00:23:24.140 | of zero occupancy ride sharing delivery.

00:23:26.740 | So the Uber that comes to you,

00:23:28.220 | as opposed to having you drive it yourself

00:23:30.480 | and it comes to you autonomously with nobody in there.

00:23:34.700 | And then you get in and drive it.

00:23:36.460 | So imagine a world where we have empty vehicles

00:23:39.980 | driving around, delivering themselves to you.

00:23:43.020 | Semi-autonomous side is

00:23:45.420 | thinking about a world where teleoperation

00:23:49.980 | plays a really crucial role,

00:23:51.380 | where it's fully autonomous

00:23:52.780 | under certain constraints in the highway,

00:23:54.220 | but a human can always step in.

00:23:56.780 | High autonomy on the highway,

00:23:58.140 | kind of like what Tesla is working towards most recently,

00:24:02.100 | is on-ramp to off-ramp.

00:24:03.940 | Now the driver is still responsible,

00:24:06.740 | liability-wise and in terms of just observing the vehicle

00:24:09.660 | and algorithmically speaking,

00:24:11.920 | but the autonomy is pretty high level

00:24:16.180 | to a point where much of the highway driving

00:24:18.180 | can be done fully autonomously.

00:24:20.260 | And low autonomy, unrestricted travel

00:24:23.580 | as an advanced driver assistance system,

00:24:27.620 | meaning that the car,

00:24:30.060 | kind of like the Tesla, the Volvo S90s,

00:24:32.820 | or the SuperCruise and the Cadillacs,

00:24:35.220 | all these kinds of L2 systems

00:24:36.940 | that are able to keep you in the lane

00:24:39.660 | 10 to 30% of the miles that you drive

00:24:42.260 | and some fraction of the time

00:24:43.660 | take some of the stress of driving off.

00:24:46.220 | And then there is some out there ideas,

00:24:48.640 | the idea of connected vehicles,

00:24:52.280 | vehicle-to-vehicle communication

00:24:54.100 | and vehicle-to-infrastructure communication,

00:24:56.420 | enabling us to navigate, for example,

00:24:59.740 | intersection efficiently without stopping,

00:25:01.700 | removing all traffic lights.

00:25:03.180 | So here shown on the bottom is our conventional approach.

00:25:06.900 | There's a queuing system that forms

00:25:09.380 | because of traffic lights that turn red, green, yellow,

00:25:11.980 | and without traffic lights

00:25:14.580 | and with communication to the infrastructure

00:25:16.220 | and between the vehicles,

00:25:17.340 | you can actually optimize that

00:25:18.620 | to significantly increase the traffic flow through a city.

00:25:22.760 | Of course, there's the boring solution

00:25:26.580 | of tunnels under cities,

00:25:32.420 | layers of tunnels under cities,

00:25:36.400 | tunnels all the way down.

00:25:40.060 | Autonomous vehicles,

00:25:43.180 | basically by the design of the tunnel,

00:25:47.500 | constraining the problem to such a degree that,

00:25:50.940 | I mean, the idea of autonomy just is completely transformed

00:25:54.960 | that you're basically,

00:25:56.160 | a car is able to transform itself into a mini train,

00:25:59.600 | into a mini public transit entity

00:26:01.880 | for a particular period of time.

00:26:04.200 | So you get into that tunnel,

00:26:05.780 | you drive at 200 miles an hour,

00:26:07.680 | and not necessarily drive, be driven 200 miles an hour,

00:26:11.920 | and then you get out of the tunnel.

00:26:14.240 | Of course, there's the flying cars,

00:26:16.160 | personalized flying car vehicles.

00:26:18.260 | I will not, I mean,

00:26:20.920 | (audio cuts out)

00:26:24.380 | Rodney, as I mentioned before,

00:26:25.720 | does believe that we'll have them in 2050.

00:26:28.380 | There's a lot of people that are seriously

00:26:30.080 | actually thinking about this problem,

00:26:32.680 | is there's a level of autonomy, obviously,

00:26:35.080 | that's required here for a regular person,

00:26:37.940 | like, I don't know,

00:26:40.400 | somebody without a pilot's license, for example,

00:26:42.900 | to be able to take off and land.

00:26:45.280 | Making that experience accessible to regular people

00:26:49.280 | means that there's going to be

00:26:50.280 | a significant amount of autonomy involved.

00:26:52.360 | One of the people really,

00:26:53.960 | one of the companies really seriously working on this

00:26:56.940 | is Uber, with the Uber Elevate, Uber Air,

00:27:00.880 | I think it's called,

00:27:02.240 | and the idea is that you would meet your vehicle

00:27:05.660 | not on the street, but at a roof,

00:27:08.200 | you take an elevator, you meet them at the roof of the,

00:27:11.000 | of a building.

00:27:13.040 | This video is from Uber,

00:27:15.120 | and they're seriously addressing this problem.

00:27:18.060 | Many of the great solutions to the world's problems

00:27:21.320 | have been laughed at at some point.

00:27:23.600 | So, let's not laugh too loud at these possibilities.

00:27:28.260 | Back in my day, we used to drive in the street.

00:27:32.600 | Okay, so,

00:27:34.160 | 10,000 vehicles,

00:27:37.600 | if that's the bar.

00:27:39.440 | I sort of out of curiosity asked,

00:27:42.020 | did a little public poll,

00:27:43.860 | 3,000 people responded,

00:27:46.120 | asked who will be first to deploy 10,000

00:27:49.520 | fully autonomous cars operating on public roads

00:27:53.240 | without a safety driver?

00:27:55.560 | And several options percolated,

00:27:59.620 | with Tesla getting 50%, 57% of the vote,

00:28:04.160 | and Waymo getting 21% of the vote,

00:28:06.800 | and 14% someone else,

00:28:09.560 | and 8% the curmudgeons and the engineers

00:28:15.200 | saying no one in the next 50 years will do it.

00:28:17.760 | And again, in 1998, when Google came along,

00:28:23.120 | the leaders of the space were Ask Jeeves

00:28:26.520 | and InfoSeek and Excite,

00:28:28.240 | all services I've used,

00:28:30.280 | and probably some people in this room have used,

00:28:32.360 | Lycos, Yahoo.

00:28:34.140 | Obviously, they were the leaders in the space,

00:28:36.720 | and Google disrupted that space completely.

00:28:39.120 | So, this poll shows the current leaders,

00:28:43.460 | but it's wide open to ideas,

00:28:45.440 | and that's why there's a lot of autonomous vehicle companies.

00:28:47.880 | Some companies are taking advantage of the hype

00:28:51.760 | and the fact that there's a lot of investment in the space,

00:28:54.220 | but some companies, like some of the speakers

00:28:58.680 | visiting this course,

00:28:59.680 | are really trying to solve this problem.

00:29:01.400 | They wanna be the next Google,

00:29:02.840 | the next billion, multi-billion,

00:29:04.800 | next trillion dollar company,

00:29:06.440 | by solving the problem.

00:29:08.020 | So, it's wide open.

00:29:09.400 | But currently, Tesla,

00:29:11.220 | with the semi-autonomous vehicle approach,

00:29:15.160 | working towards trying to become fully autonomous,

00:29:18.700 | and Waymo, starting with the fully autonomous,

00:29:20.780 | working towards achieving scale at the fully autonomous,

00:29:23.640 | are the leaders in the space.

00:29:25.760 | Given that

00:29:26.940 | ranking in 2019,

00:29:31.140 | let's take a quick step back to 2005

00:29:33.500 | with the DARPA challenge,

00:29:34.620 | when the story began.

00:29:36.500 | The race to the desert,

00:29:38.140 | when Stanley from Stanford won a race to the desert.

00:29:42.200 | That really captivated people's imagination

00:29:44.420 | about what's possible.

00:29:45.620 | And a lot of people have said

00:29:46.980 | that the autonomous vehicle problem is solved in 2005.

00:29:50.340 | They really said,

00:29:51.240 | the idea was, especially because in 2004,

00:29:54.300 | nobody finished that race.

00:29:55.740 | 2005, four cars finished the race.

00:29:58.140 | It was like, well, we cracked it.

00:29:59.700 | This is it.

00:30:01.220 | And then some critics said that

00:30:04.580 | urban driving is really nothing comparable

00:30:07.580 | to desert driving.

00:30:09.180 | Desert is very simple.

00:30:10.240 | There's no obstacles and so on.

00:30:11.780 | It's really a mechanical engineering problem.

00:30:13.660 | It's not a software problem.

00:30:14.740 | It's not a fundamentally,

00:30:16.700 | it's not really an autonomous driving problem

00:30:18.460 | as it would be delivered to consumers.

00:30:20.420 | And then, of course, in 2007,

00:30:22.020 | the DARPA put together the Urban Grand Challenge,

00:30:24.540 | and several people finished that,

00:30:26.560 | with CMU's boss winning.

00:30:28.500 | And so,

00:30:30.260 | the thought was, at that point, that's it, we're done.

00:30:33.220 | As Ernest Rutherford, a physicist, said,

00:30:37.180 | that physics is the only real science.

00:30:38.940 | The rest is just stamp collecting.

00:30:40.420 | All the biology, chemistry, certainly,

00:30:42.740 | boy, I wouldn't wanna know

00:30:43.660 | what he thinks about computer science.

00:30:46.140 | It's just all the stupid, silly details.

00:30:48.260 | Physics is the fundamentals.

00:30:49.500 | And that was the idea,

00:30:51.460 | with the DARPA Grand Challenge and solving that,

00:30:53.620 | that we solve the fundamental problem of autonomy.

00:30:56.400 | And the rest is just for industry

00:30:59.140 | to figure out some of the details

00:31:00.260 | of how to make an app

00:31:01.980 | and make a business out of it.

00:31:04.140 | So that could be true.

00:31:06.180 | And the underlying beliefs there

00:31:07.620 | is that driving is an easy task,

00:31:09.700 | that it's solvable.

00:31:12.100 | The thing that we do as human beings,

00:31:13.660 | that it's pretty formalizable,

00:31:15.980 | it's pretty easy to solve with autonomy,

00:31:19.900 | that the other idea is that humans are bad at driving.

00:31:22.980 | This is a common belief.

00:31:24.500 | Not me, not you, but everybody else.

00:31:27.580 | Nobody in this room,

00:31:28.480 | but everybody else is a terrible driver.

00:31:30.460 | The kind of intuition that we have

00:31:32.540 | about our experience in traffic

00:31:34.000 | leads us to believe that humans

00:31:35.180 | are just really bad at driving.

00:31:37.020 | And from the human factors psychology side,

00:31:40.800 | there's been over 70 years of research,

00:31:45.800 | showing that humans are not able to monitor,

00:31:54.220 | maintain vigilance, monitoring a system.

00:31:57.180 | So when you put a human in a room with a robot

00:32:00.540 | and say, "Watch that robot,"

00:32:03.340 | they start texting like 15 seconds in.

00:32:07.800 | So that's the fundamental psychology.

00:32:09.880 | There's thousands of papers on this.

00:32:11.840 | People are, they tune out, they overtrust the system,

00:32:14.700 | they misinterpret the system,

00:32:16.700 | and they lose vigilance.

00:32:19.080 | Those are the three underlying beliefs.

00:32:21.080 | It very well could be true,

00:32:22.700 | but what if it is not?

00:32:25.740 | So we have to consider that it is not.

00:32:28.620 | The driving task is easy,

00:32:30.380 | because if you think the driving task

00:32:31.780 | is easy and formalizable and solvable

00:32:33.500 | by autonomous vehicles,

00:32:34.400 | you have to solve this problem.

00:32:36.420 | The subtle vehicle to vehicle,

00:32:38.900 | vehicle to pedestrian nonverbal communication

00:32:41.260 | that happens here in a dramatic sense,

00:32:44.780 | but really happens in a subtle sense,

00:32:47.180 | millions of times every single day in Boston.

00:32:50.740 | Subtle nonverbal communication between vehicles.

00:32:53.860 | You go, no you go.

00:32:55.360 | You have to solve all the crazy road conditions

00:33:00.100 | where in a split seconds,

00:33:01.300 | you have to make a decision about,

00:33:04.780 | so in snowy, icy weather, rain,

00:33:07.900 | limited visibility conditions,

00:33:09.620 | you have 100, 200 milliseconds to make a decision.

00:33:13.220 | Your algorithm based on the perception

00:33:14.900 | has to make a control decision.

00:33:16.460 | Then you have to deal with the nonverbal communication

00:33:20.980 | with pedestrians.

00:33:22.380 | These unreasonable, irrational creatures,

00:33:25.340 | us human beings.

00:33:26.780 | You have to not only understand what they're,

00:33:30.260 | the intent of the movement that's anticipated.

00:33:35.180 | So anticipating the trajectory of the pedestrian,

00:33:37.180 | you also have to assert yourself in a game theoretic way.

00:33:40.420 | As crazy as it might sound,

00:33:42.060 | you have to threaten yourself,

00:33:43.140 | you have to take a risk.

00:33:45.020 | You have to take a risk that if I don't slow down,

00:33:48.280 | like that ambulance didn't slow down,

00:33:50.300 | that the pedestrian will slow down.

00:33:52.800 | Algorithmically, we're afraid to do that.

00:33:57.920 | The idea that a pedestrian that's moving,

00:34:02.320 | we anticipate their trajectory

00:34:03.660 | based on the simple physics of the current velocity,

00:34:06.100 | the momentum, they're gonna keep going

00:34:08.020 | with some probability.

00:34:09.420 | The fact that by us accelerating,

00:34:12.600 | we might make that pedestrian stop

00:34:15.040 | is something that we have to incorporate into algorithms

00:34:17.260 | and we don't today.

00:34:18.720 | So that, and we don't know how to really.

00:34:21.160 | So if driving is easy, we have to solve that too.

00:34:24.580 | And of course, the thing I showed yesterday

00:34:26.320 | with the coast runners and the boat going around

00:34:28.820 | and all the ethical dilemmas

00:34:35.160 | from the moral machine to the more

00:34:37.940 | serious engineering aspects

00:34:39.900 | that from the unintended consequences

00:34:43.460 | that arise from having to formalize

00:34:48.340 | the objective function under which

00:34:50.300 | a planning algorithm operates.

00:34:52.820 | If there's any learning that as I showed yesterday,

00:34:55.960 | a boat on the left driven by a human

00:34:57.980 | wants to finish the race,

00:34:59.420 | the boat on the right figures out

00:35:00.860 | that it doesn't have to finish the race,

00:35:02.420 | it can pick up turbos along the way

00:35:04.660 | and gets much more reward.

00:35:06.620 | So if the objective function is to maximize the reward,

00:35:09.060 | you can slam into the wall over and over and over again,

00:35:11.500 | and that's actually the way to optimize the reward.

00:35:14.420 | And those are the unintended consequences

00:35:16.820 | of an algorithm that has to be formalizable

00:35:20.100 | to the objective function without a human in the loop.

00:35:22.700 | Humans are bad at driving.

00:35:24.900 | As I showed yesterday,

00:35:26.320 | humans, if they're bad at anything,

00:35:30.800 | it's about having a good intuition about

00:35:33.080 | what's hard and what's easy.

00:35:36.280 | The fact that we have 540 million years worth of data

00:35:38.800 | on our visual perception system

00:35:40.720 | means we don't understand how damn impressive it is

00:35:43.920 | to be able to perceive and understand the scene

00:35:46.120 | in a split second, maintain context,

00:35:48.980 | maintain an understanding of performing

00:35:51.720 | all the visual localization tasks

00:35:54.560 | about anticipating the physics of the scene and so on.

00:35:58.100 | And then there's a control side.

00:36:01.020 | The humans don't give enough credit to ourselves.

00:36:04.100 | We're incredible.

00:36:05.460 | A state of the art soccer player on the left

00:36:09.700 | and a state of the art robot on the right.

00:36:13.960 | (audience laughing)

00:36:16.960 | I think there's like four or five times he scores.

00:36:30.240 | All right.

00:36:31.640 | And that's all the movement and so on involved with that.

00:36:36.640 | Of course here, that's the human robot.

00:36:39.520 | That's a really incredible work that's done

00:36:41.720 | for the DARPA Robotics Challenge

00:36:43.360 | with the human robots on the right

00:36:45.340 | and incredible work by the human people

00:36:50.340 | doing the same kind of tasks,

00:36:53.180 | much more impressive tasks I would say.

00:36:55.040 | So that's where we stand.

00:36:57.280 | And the ones on the right are actually not fully autonomous.

00:36:59.560 | There's still some human in the loop.

00:37:01.120 | There's just noisy broken communication.

00:37:03.560 | So humans are incredible

00:37:05.240 | in terms of our ability to understand the world

00:37:07.680 | and in terms of our ability to act in that world.

00:37:10.360 | And the fact that humans, the idea, the view,

00:37:14.400 | the popular view grounded in the psychology

00:37:17.000 | that humans and automations don't mix well,

00:37:20.800 | over trust, misunderstanding,

00:37:22.520 | loss of vigilance, decrement and so on.

00:37:24.660 | That's not an obvious fact.

00:37:27.640 | It happens a lot in the lab.

00:37:29.080 | Most of the experiments are actually in the lab.

00:37:31.860 | This is the difference.

00:37:32.700 | You put many of you,

00:37:34.900 | you put a undergrad, grad student in a lab and say,

00:37:39.520 | "Here, watch this screen and wait for the dot to appear."

00:37:44.080 | They'll tune out immediately.

00:37:45.740 | But when it's your life and you're on the road,

00:37:48.440 | it's just you in the car, it's a different experience.

00:37:51.640 | It's not completely obvious the vigilance will be lost.

00:37:54.480 | And it's not a complete,

00:37:55.680 | when it's just you and the robot,

00:37:58.480 | it's not completely obvious what the psychology,

00:38:02.040 | what the attentional mechanism,

00:38:03.400 | what the vigilance there looks like.

00:38:05.240 | So one of the things we did is we instrumented here

00:38:07.360 | 22 Teslas and observe people now over a period of two years

00:38:11.320 | of what they actually do when they're driving on a pilot,

00:38:14.320 | driving these systems.

00:38:15.540 | In red shown manually controlled vehicles

00:38:17.500 | and cyan showed vehicle control on autopilot.

00:38:22.000 | Now there's a lot of details here

00:38:23.600 | and we have a lot of presentation on this,

00:38:25.280 | but really the fundamentals are,

00:38:26.880 | is that they drive 34%,

00:38:29.800 | large percentage of the miles in autopilot.

00:38:32.660 | And in 26,000 moments of transfer of control,

00:38:37.660 | they are always vigilant.

00:38:42.240 | There's not a moment once in this dataset

00:38:45.760 | where they respond too late

00:38:47.260 | to a critical situation, to a challenging road situation.

00:38:52.620 | Now the dataset, 22 vehicles,

00:38:55.720 | that's 0.1% or less than the full Tesla fleet

00:39:01.540 | that has autopilot, but it's still an inkling.

00:39:04.420 | It's not obvious that it's not possible

00:39:06.100 | to build a system that works together with a human being.

00:39:08.940 | And that system essentially looks like this.

00:39:14.020 | Some percentage, 90%, maybe less, maybe more.

00:39:17.420 | When it can solve the problem of autonomous driving,

00:39:21.120 | it solves it and when it needs human help,

00:39:22.940 | it asks for help.

00:39:24.380 | That's the trade-off, that's the balance.

00:39:27.100 | On the fully autonomous side, on the right,

00:39:29.420 | it has to solve here with citations.

00:39:32.300 | And there's references always on the bottom.

00:39:35.140 | All the problems have to be solved exceptionally,

00:39:37.500 | perfectly, from mapping localization

00:39:40.420 | to the scene perception, to control, to planning,

00:39:43.620 | to being able to find a safe harbor at any moment,

00:39:47.880 | to also being able to do external HMI,

00:39:50.380 | communication with the other pedestrians

00:39:51.820 | and vehicles in the scene.

00:39:53.140 | And then there's teleoperation,

00:39:54.540 | vehicle to vehicle, vehicle to eye.

00:39:56.340 | You have to solve those perfectly

00:39:57.980 | if you want to solve the fully autonomous problem.

00:40:00.460 | As I said, including all the crazy things

00:40:02.660 | that happen in driving.

00:40:04.380 | And if you approach the shared autonomy side,

00:40:06.860 | the semi-autonomous, where you're only responsible

00:40:09.620 | for a large percentage, but not 100% of the driving,

00:40:12.280 | then you have to solve the human side,

00:40:14.640 | the human interaction, the sensing what the driver is doing,

00:40:18.100 | the collaborating, communicating with the driver,

00:40:20.800 | and the personalization aspect that learns with the driver.

00:40:26.020 | Like we've, as I said, you can go online,

00:40:28.220 | we have a lot of demonstrations of these kinds of ideas,

00:40:30.700 | but the natural language, the communication,

00:40:33.680 | I think is critical for all of us,

00:40:35.420 | as we're tweeting, as all of us do.

00:40:37.320 | So it's as simple as, so this is just demonstration

00:40:47.860 | of a vehicle taking control when the attention over time,

00:40:52.020 | the driver is being,

00:40:54.700 | (man speaking off mic)

00:40:56.860 | Okay, we got it, thank you.

00:40:58.220 | Okay, so basically, a smartphone use,

00:41:05.300 | which has gone up year by year,

00:41:06.620 | and we're doing a lot of analysis on that,

00:41:08.180 | it's really what people do in the car,

00:41:10.100 | is they use their phone.

00:41:11.540 | Whether it's manual or autonomous driving,

00:41:14.460 | or semi-autonomous driving.

00:41:15.600 | So being able to manage that,

00:41:17.920 | to communicate with the driver about

00:41:20.200 | when they should be paying attention,

00:41:22.540 | which may not be always,

00:41:23.980 | you're sort of balancing the time,

00:41:25.660 | when is it a critical time to pay attention,

00:41:27.660 | when it's not, and communicating effectively,

00:41:30.420 | learning with the driver,

00:41:31.580 | that problem is a fundamental machine learning problem.

00:41:35.320 | There's a lot of data, visible light,

00:41:37.580 | everything about the driver,

00:41:38.900 | and it's a psychology problem.

00:41:40.620 | So we have data, we have complicated human beings,

00:41:44.700 | and it's a human-robot interaction problem

00:41:47.360 | that deserves solving.

00:41:49.540 | But as you'll hear,

00:41:51.300 | on the beyond the human side,

00:41:54.380 | looking out into the world,

00:41:56.060 | people that are trying to solve

00:41:57.020 | the fully autonomous vehicle,

00:41:58.120 | it's really a two approach consideration.

00:42:00.880 | One approach is vision, cameras,

00:42:05.180 | and deep learning, right?

00:42:07.660 | Collect a huge amount of data.

00:42:09.940 | So cameras have this aspect that they,

00:42:13.580 | they're the highest resolution of information available.

00:42:16.260 | It's rich texture information.

00:42:18.700 | And there's a lot of it,

00:42:20.020 | which is exactly what neural networks love, right?

00:42:22.920 | So to be able to cover all the crazy edge cases,

00:42:26.220 | vision data, camera data, visible light data

00:42:30.580 | is the exactly the kind of data you need

00:42:32.360 | to collect a huge amount of,

00:42:33.520 | to be able to generalize over all the crazy,

00:42:35.940 | countless edge cases that happen.

00:42:38.260 | It's also feasible, all the major data sets,

00:42:40.740 | all the, in terms of cost, interest, scale,

00:42:45.340 | all the major data sets of visible light cameras.

00:42:48.660 | That's another pro and they're cheap.

00:42:50.640 | And the world as it happens,

00:42:53.340 | whoever designed the simulation that we're all living in,

00:42:56.740 | made it such that our,

00:42:58.660 | our world, our roads and our world is designed

00:43:03.620 | for human eyes.

00:43:05.140 | So eyes is the way we perceive the world.

00:43:09.540 | And so the landmark is also on is visual,

00:43:11.660 | most of the road textures that you use to navigate,

00:43:17.060 | to drive are visible, are made for human eyes.

00:43:21.500 | The cons are that without a ton of data,

00:43:24.780 | and we don't know how much,

00:43:26.180 | they're not accurate.

00:43:29.020 | You make errors because driving is ultimately

00:43:31.340 | about 99.999999% accuracy.

00:43:34.100 | And so that's what I mean by not accurate.

00:43:37.220 | It's really difficult to reach that level.

00:43:40.340 | And then the second approach is LiDAR,

00:43:45.140 | taking a very particular constrained set of roads,

00:43:48.820 | mapping the heck out of them,

00:43:51.060 | understanding them fully

00:43:52.100 | at a different weather condition and so on,

00:43:53.900 | and then using the most accurate sensors available,

00:43:57.980 | a Swedish sensors, but really LiDAR at the forefront,

00:44:01.580 | being able to localize yourself effectively.

00:44:03.860 | The pros there that it's consistent,

00:44:06.540 | especially when machine learning is not evolved,

00:44:09.060 | it's consistent and reliable.

00:44:11.360 | And it's explainable.

00:44:12.620 | If it fails, you can understand why,

00:44:14.900 | you can account for those situations.

00:44:16.620 | It's not so much true for machine learning methods.

00:44:19.900 | It's not so much explainable

00:44:21.140 | why it failed in a particular situation.

00:44:23.940 | The accuracy is higher as we'll talk about.

00:44:26.300 | The cons of LiDAR is that it's expensive.

00:44:29.280 | And most of the approaches

00:44:31.700 | in perceiving the world using LiDAR primarily

00:44:35.620 | are not deep learning based.

00:44:37.220 | And therefore they're not learning over time.

00:44:39.500 | And if they were deep learning based,

00:44:41.180 | there's a reason they're not,

00:44:42.340 | it's 'cause you need a lot of car,

00:44:45.660 | you need a lot of LiDAR data.

00:44:47.820 | And there's only a tiny percentage of cars in the world,

00:44:51.580 | quite obviously, are equipped with LiDAR

00:44:55.560 | in order to collect that data.

00:44:57.860 | So quickly running through the sensors,

00:45:00.860 | radar is the,

00:45:04.140 | it's kind of like the offensive line of football.

00:45:06.900 | They're actually the ones that do all the work

00:45:08.660 | and they never get the credit.

00:45:10.900 | So radar is that.

00:45:12.580 | It's always behind to catch,

00:45:14.780 | to actually do the detection in terms of obstacle,

00:45:17.780 | the most critical, safety critical obstacle avoidance.

00:45:20.860 | It's cheap, it does extremely well,

00:45:23.820 | it does well in extreme weather,

00:45:25.780 | but it's low resolution.

00:45:27.740 | So it's cannot stand on its own

00:45:30.660 | to achieve any kind of degree of high autonomy.

00:45:33.300 | Now on the lighter side, it's expensive.

00:45:35.540 | It's extremely accurate depth information,

00:45:37.420 | 3D cloud, point cloud information.

00:45:39.820 | Its resolution is much higher than radar,

00:45:42.700 | but still lower than visible light.

00:45:45.020 | And there is, depending on the sensor,

00:45:47.260 | 360 degree visibility that's built in.

00:45:50.500 | So there's a difference in resolution here,

00:45:53.700 | visualized LiDAR on the right, radar on the left.

00:45:57.900 | The resolution is just much higher and is improving

00:46:00.220 | and the cost is going down and so on.

00:46:02.100 | Now on the camera side, it's cheap.

00:46:04.380 | Everybody got one.

00:46:05.700 | The resolution is extremely high

00:46:07.100 | in terms of the amount of information transferred per frame.

00:46:11.540 | And everybody, you know, really the scale of,

00:46:16.540 | the number of vehicles that have this equipped is humongous.

00:46:19.860 | So it's ripe for application of deep learning.

00:46:24.860 | And the challenge is it's noisy,

00:46:27.820 | it's bad at depth estimation,

00:46:29.460 | and it's not good in extreme weather.

00:46:35.860 | So if we kind of use this plot to look,

00:46:38.820 | to compare these sensors,

00:46:40.420 | to compare these different approaches.

00:46:42.540 | So LiDAR works in the dark, variable lighting conditions,

00:46:47.180 | has pretty good resolution, has pretty good range,

00:46:50.100 | but it's expensive, it's huge,

00:46:55.100 | and it doesn't provide rich textural contrast information.

00:47:00.900 | And it's also sensitive to fog and rain conditions.

00:47:04.460 | Now, ultrasonic sensors catch a lot of those problems.

00:47:08.180 | They're better at detecting proximity.

00:47:09.900 | They're high resolution in objects that are close,

00:47:13.420 | which is why they're often used for parking,

00:47:15.060 | but they can still also be integrated

00:47:16.740 | in the sensor fusion package for an autonomous vehicle.

00:47:20.740 | They really catch a lot of the problems that radar has.

00:47:25.140 | They complement each other well.

00:47:27.340 | And radar, cheap, tiny, detect speed,

00:47:34.020 | and has pretty good range,

00:47:38.340 | but has terrible resolution.

00:47:42.300 | There's very little information being provided.

00:47:44.660 | And then cameras, a lot of rich information.

00:47:50.620 | They're cheap, they're small, range is great.

00:47:53.620 | The best range actually of all the sensors,

00:47:56.260 | and works in bright conditions,

00:47:59.780 | but doesn't work in the dark in extreme conditions,

00:48:01.900 | and it's just susceptible to all these kinds of problems.

00:48:04.940 | And doesn't detect speed,

00:48:06.260 | unless you do some tricky structure

00:48:09.260 | from motion kind of things.

00:48:10.580 | So here's where sensor fusion steps in,

00:48:12.780 | and everybody works together to build an entire picture.

00:48:17.780 | That's how this plot works.

00:48:19.260 | You can stack it on top of each other.

00:48:20.980 | So if you look at a suite that, for example,

00:48:22.660 | Tesla is using, which is ultrasonic, radar, and camera,

00:48:25.940 | and you compare it to just LiDAR,

00:48:27.940 | and see how these paths compare,

00:48:29.900 | that actually the suite of camera, radar, and ultrasonic

00:48:34.900 | are comparable to LiDAR.

00:48:37.100 | So those are the two comparisons that we have.

00:48:40.580 | You have the costly non-machine learning way of LiDAR,

00:48:44.540 | and you have the cheap, but needs a lot of data,

00:48:49.540 | and is not explainable or reliable

00:48:51.940 | in the near term vision-based approach.

00:48:54.740 | And those are the two competing approaches.

00:48:56.460 | Now, of course, way more, so we'll talk about

00:48:58.180 | they're trying to use both,

00:48:59.500 | but ultimately the question is, who catches,

00:49:03.740 | who is the fail safe?

00:49:07.860 | In the semi-autonomous way,

00:49:09.660 | when there's a camera-based method,

00:49:11.020 | the human is the fail safe.

00:49:12.900 | When you say, oh, crap, I don't know what to do,

00:49:15.820 | the human catches.

00:49:17.660 | In the fully autonomous mode,

00:49:20.500 | so what Waymo's working on, and others,

00:49:24.000 | the fail safe is LiDAR.

00:49:26.140 | Fail safe is maps, that you can't rely on the human,

00:49:30.540 | but you know this road so well,

00:49:32.660 | that if the cameras freak out,

00:49:34.260 | if there's any of the sensors freak out,

00:49:36.180 | that you're able to, you have such good maps,

00:49:39.460 | you have such good accurate sensors,

00:49:41.220 | that the fundamental problem of obstacle avoidance,

00:49:44.060 | which is what safety is about, can be solved.

00:49:49.060 | The question is, what kind of experience that creates.

00:49:52.380 | In the meantime, as the people debate,

00:49:54.540 | try to make money, start companies,

00:49:56.860 | there's just lots of data.

00:49:58.620 | Ford F-150 is still the most popular car in America.

00:50:03.200 | Manually driven cars are still happening,

00:50:04.900 | so there's a lot of data happening.

00:50:06.220 | Semi-autonomous cars, every company is now releasing

00:50:10.300 | more and more semi-autonomous technology,

00:50:13.180 | so that's all data.

00:50:15.420 | And what that boils down to,

00:50:16.900 | is the two paths they're walking towards,

00:50:20.060 | is vision versus LiDAR, L2 versus L4,

00:50:23.460 | semi-autonomous versus fully autonomous.

00:50:25.820 | Tesla on the semi-autonomous front,

00:50:27.660 | has reached one billion miles,

00:50:29.300 | Waymo the leader on the autonomous front,

00:50:31.180 | has reached 10 million miles.

00:50:33.100 | The pros and cons that I've outlined them.

00:50:35.500 | One, the vision one, the one I'm obviously

00:50:38.740 | very excited about, and machine learning researchers

00:50:42.140 | are excited about, which fundamentally relies on huge data

00:50:45.220 | and deep learning.

00:50:46.780 | The neural networks that are running inside the Tesla,

00:50:51.600 | and with their new, it's kind of the same kind of path

00:50:55.340 | as Google is taking from the GPU to the GPU.

00:50:58.500 | Tesla is taking from the NVIDIA Drive PX2 system,

00:51:02.020 | sort of more general GPU based system

00:51:03.980 | to creating their own ASIC,

00:51:05.340 | and having a ton of awesome neural networks

00:51:07.580 | running on their car.

00:51:08.580 | That kind of path that others are beginning to embrace,

00:51:11.220 | is really interesting to think about

00:51:13.720 | for machine learning engineers.

00:51:15.020 | And then people that are maybe more grounded,

00:51:19.020 | and actually wanna really value safety, reliability,

00:51:24.020 | and sort of from the automotive world,

00:51:27.300 | are thinking well we need,

00:51:28.440 | machine learning is not explainable,

00:51:30.220 | it's difficult to work with, it's not reliable,

00:51:35.220 | and so in that sense we have to have a sensor suite

00:51:37.880 | that is extremely reliable.

00:51:38.800 | Those are the two paths.

00:51:40.040 | Yep, question.

00:51:43.360 | The question is, there's all kinds of things

00:51:47.900 | you need to perceive.

00:51:48.960 | Stop signs and traffic lights, pedestrians and so on.

00:51:52.040 | Some of them, if you hit them, it's a problem.

00:51:55.480 | Some of them are a bag flying through the air,

00:51:58.540 | and all have different visual characteristics,

00:52:00.360 | all have different characteristics

00:52:01.600 | for all the different sensors.

00:52:03.540 | So LIDAR can detect solid body objects,

00:52:08.540 | camera is better at detecting.

00:52:10.480 | As last year, Sasha Arnoud talked about,

00:52:14.280 | I think fog or smoke, these are interesting things.

00:52:17.840 | They might look like an object to certain sensors

00:52:20.200 | and not to others, but the traffic light detection problem,

00:52:25.200 | luckily with cameras, is pretty solved at this point.

00:52:30.640 | So that's luckily the easy part.

00:52:33.880 | The hard part is when you have a green light,

00:52:36.080 | and there's a drunk, drug, drowsy, or distracted,

00:52:41.080 | the four Ds that Nitz outlined, pedestrian,

00:52:43.280 | trying to cross, what to do.

00:52:46.240 | That's the hard part.

00:52:47.660 | So the road ahead for us, as engineers, as scientists,

00:52:51.600 | the thing I'm super excited about,

00:52:53.440 | the possibility of artificial intelligence

00:52:55.080 | having a huge impact, is taking this step

00:52:57.200 | from having these, even if they're large, toy data sets,

00:53:02.200 | toy problems, toy benchmarks,

00:53:07.260 | of ImageNet classification in Cocoa,

00:53:10.640 | all the exciting deep RL stuff that we'll talk about

00:53:13.520 | in the future weeks, really are toy examples.

00:53:16.440 | The game of Go and chess and so on.

00:53:18.300 | But taking those algorithms and putting them in cars

00:53:20.900 | where they can save people's lives,

00:53:22.680 | and they actually directly touch and impact

00:53:25.660 | our entire civilization.

00:53:27.500 | That's actually the defining problem

00:53:29.180 | for artificial intelligence in the 21st century,

00:53:31.620 | is AI that touches people in a real way.

00:53:34.900 | And I think cars, autonomous vehicles,

00:53:36.980 | is one of the big ways that that happens.

00:53:39.400 | We get to deal with the psychology, the philosophy,

00:53:42.660 | the sociology aspects of it,

00:53:44.420 | how we socially think about it, to the robotics problem,

00:53:47.500 | to the perception problem.

00:53:48.780 | It's a fascinating space to explore.

00:53:50.940 | And we have many guest speakers exploring that

00:53:54.780 | in different ways, and that's really exciting

00:53:57.900 | to see how these people are trying to change the world.

00:54:01.220 | So with that, I'd like to thank you very much.

00:54:04.660 | Go to deeplearning.mit.edu,

00:54:07.320 | and the code is always available online.

00:54:10.020 | (audience applauding)

00:54:13.180 | (static crackling)

00:54:16.100 | (static crackling)

00:54:19.020 | (static crackling)

00:54:21.940 | (static crackling)

00:54:24.860 | (static crackling)

00:54:27.780 | (static crackling)

00:54:30.700 | Thanks for watching.

Self-Driving Cars: State of the Art (2019)

Chapters