Automated Synchronization of Driving Data: Video, Audio, IMU, and Telemetry
00:00:00.000 | [car driving]
00:00:07.000 | This is a quick demo of how car vibration and steering events can be used to synchronize driving data.
00:00:12.000 | The video itself is a visualization of the data streams we're working with.
00:00:17.000 | The audio you're hearing in the background, besides my voice, is from a shotgun microphone placed behind the rear right tire.
00:00:25.000 | The middle column has three images, each from a different webcam.
00:00:30.000 | Front, dashboard, and face.
00:00:33.000 | The dashboard video has an overlaid steering wheel icon that is visualizing the position of the steering wheel as supported by the CAN network.
00:00:41.000 | The top left image shows the dense optical flow in the video of the forward roadway.
00:00:49.000 | The bottom left just shows our location on a map. We're in beautiful Cambridge, Massachusetts.
00:00:57.000 | And the rest are plots showing the ten second window around the current measurement of various sensors.
00:01:05.000 | On the left are the horizontal optical flow in the front video and the steering wheel position.
00:01:13.000 | On the right are the audio energy from the shotgun microphone, the Y component of the optical flow from the three webcams, and finally the Z axis of the accelerometer.
00:01:27.000 | What we would like to do is to synchronize all of these sensors, either online or offline as a post-processing step.
00:01:35.000 | We do this by first synchronizing the video of the forward roadway with the CAN network by looking at steering events.
00:01:45.000 | When you make a turn, like the one coming up here, the horizontal optical flow will be negative if it's a right turn and positive if it's a left turn.
00:01:56.000 | Coming up here is a left turn and you will see in the top left image the dense optical flow will light up all the same color.
00:02:05.000 | It will be a positive value since it's a left turn.
00:02:10.000 | We can then determine the optimal shift for the synchronization between the steering wheel and the forward video by computing the cross correlation function,
00:02:21.000 | the maximum value for the cross correlation function, to determine the shift.
00:02:27.000 | In the same way, we synchronize the rest of the sensors with the video of the forward roadway using vibration events.
00:02:36.000 | On the right are five plots showing the audio energy, the Y component of the optical flow for the three webcams, and the Z axis of the accelerometer,
00:02:45.000 | each capturing the vibration of the car caused by the road.
00:02:49.000 | A few examples are coming up shortly here.
00:02:52.000 | So steering and vibration gives us a signal that we can use for passive synchronization.
00:03:17.000 | The result is a synchronized data set which is important both for the analysis of driver behavior and for the design of ADAS systems that use decision fusion to make real-time prediction based on multiple sensor streams.
00:03:31.000 | The paper along with a sample data set and source code are available in the description.