This is the Human-Centered Autonomous Vehicle. One of the main ideas underlying our work is that solving the task of autonomous driving is more complicated and more fascinating than the strictly robotics challenges of localization, mapping, perception, control, and planning. You also have to enable the vehicle to perceive, predict, communicate, and collaborate with human beings.

The humans inside the car, like the driver and the passengers, and the humans outside the car, like the pedestrians, cyclists, the drivers of other vehicles, and even teleoperators. The studies, the code, the data, and the demos we release all consider autonomous driving in this kind of human-centered way, where the control is transferred from human to machine and back to human based on the state of the external driving environment and the state of the driver.

What we'd like to demonstrate today is the basics, voice-based transfer of control from human to machine based on whether the driver is paying attention to the road or not. Inside, we have two cameras on the driver, one on the driver's face, one on the driver's body. We have two cameras looking at the external roadway, and we have a few other cameras for filming purposes.

There's a center stack display showing who's in control of the vehicle, human or machine. So currently, the human is in control of the vehicle. Let's drive. Split the car and drive. On the center stack display, it shows the gear as drive. The perception, control, and driver state sensing algorithms you see today are running in real time, but the visualizations you're seeing in video are done in offline post-processing.

Our perception system today is vision-based using two neural networks. One is doing road segmentation, the other is doing object detection of vehicles, cyclists, pedestrians, traffic signs, traffic lights. The acceleration, braking, and steering of the car is performed by PID controllers. The driver state sensing that we're showing today is glance region classification, and that's performed using 3D convolutional neural networks.

High-level planning decisions to transfer control or to stop the vehicle are performed by a decision fusion algorithm that combines risk factors in the external environment and driver state, whether the driver's paying attention to the road or not. Safety for us is the number one priority, always. We are on a test track.

The vehicles and pedestrians here today are all part of our team, all part of the demonstration. There's another safety driver in the car that can stop the vehicle at any moment by pressing a single button. Okay, let's engage in a distracting activity, Twitter, and let's send a tweet. (silence) I'm typing this tweet while driving in the MIT city center.

In the MIT semi-autonomous vehicle on a test track. (silence) A test track. Lex, you appear distracted. Would you like me to take over? Yes, please. Great. I am taking control of steering and braking. (silence) The car is now in control as the center stack display shows. So I will continue with the tweet.

The car knows that I'm not paying attention and has taken control after asking me nicely for it. (silence) Video out tomorrow. Okay, here goes nothing. It's posted. Very well might be the first tweet ever sent from an autonomous vehicle while it's driving itself. (silence) (car engine roaring) Elevated driving risk detected.

I am stopping for a pedestrian. Lex, pedestrian is blocking our lane of travel. Should I honk? No, please shift gear to park. Great. We are now in park. (silence) That was a demo of the basics. Perception, motion planning, driver state sensing, transfer control and tweeting. And we have a lot more to explore together.

Our team is working on various aspects of human centered artificial intelligence toward our mission to save lives through effective human robot collaboration. (silence) (silence) (silence) (silence) (silence) (silence) (silence) (silence)