This is a quick demo of how car vibration and steering events can be used to synchronize driving data. The video itself is a visualization of the data streams we're working with. The audio you're hearing in the background, besides my voice, is from a shotgun microphone placed behind the rear right tire.

The middle column has three images, each from a different webcam. Front, dashboard, and face. The dashboard video has an overlaid steering wheel icon that is visualizing the position of the steering wheel as supported by the CAN network. The top left image shows the dense optical flow in the video of the forward roadway.

The bottom left just shows our location on a map. We're in beautiful Cambridge, Massachusetts. And the rest are plots showing the ten second window around the current measurement of various sensors. On the left are the horizontal optical flow in the front video and the steering wheel position. On the right are the audio energy from the shotgun microphone, the Y component of the optical flow from the three webcams, and finally the Z axis of the accelerometer.

What we would like to do is to synchronize all of these sensors, either online or offline as a post-processing step. We do this by first synchronizing the video of the forward roadway with the CAN network by looking at steering events. When you make a turn, like the one coming up here, the horizontal optical flow will be negative if it's a right turn and positive if it's a left turn.

Coming up here is a left turn and you will see in the top left image the dense optical flow will light up all the same color. It will be a positive value since it's a left turn. We can then determine the optimal shift for the synchronization between the steering wheel and the forward video by computing the cross correlation function, the maximum value for the cross correlation function, to determine the shift.

In the same way, we synchronize the rest of the sensors with the video of the forward roadway using vibration events. On the right are five plots showing the audio energy, the Y component of the optical flow for the three webcams, and the Z axis of the accelerometer, each capturing the vibration of the car caused by the road.

A few examples are coming up shortly here. So steering and vibration gives us a signal that we can use for passive synchronization. The result is a synchronized data set which is important both for the analysis of driver behavior and for the design of ADAS systems that use decision fusion to make real-time prediction based on multiple sensor streams.

The paper along with a sample data set and source code are available in the description.