End-to-End Learning of Driving Models with Surround-View Cameras and Route Planners

Abstract

For human drivers, having rear and side-view mirrors is vital for safe driving. They deliver a more complete view of what is happening around the car. Human drivers also heavily exploit their mental map for navigation. Nonetheless, several methods have been published that learn driving models with only a front-facing camera and without a route planner. This lack of information renders the self-driving task quite intractable. We investigate the problem in a more realistic setting, which consists of a surround-view camera system with eight cameras, a route planner, and a CAN bus reader. In particular, we develop a sensor setup that provides data for a 360-degree view of the area surrounding the vehicle, the driving route to the destination, and low-level driving maneuvers (e.g. steering angle and speed) by human drivers. With such a sensor setup we collect a new driving dataset, covering diverse driving scenarios and varying weather/illumination conditions. Finally, we learn a novel driving model by integrating information from the surround-view cameras and the route planner. Two route planners are exploited: 1) by representing the planned routes on Open-StreetMap as a stack of GPS coordinates, and 2) by rendering the planned routes on TomTom Go Mobile and recording the progression into a video. Our experiments show that: 1) 360-degree surround-view cameras help avoid failures made with a single front-view camera, in particular for city driving and intersection scenarios; and 2) route planners help the driving task significantly, especially for steering angle prediction. Code, data and more visual results will be made available at http://www.vision.ee.ethz.ch/˜heckers/Drive360. Keywords: Autonomous driving · end-to-end learning of driving · route planning for driving · surround-view cameras · driving dataset

A survey of deep learning techniques for autonomous driving

“…End2End control papers mainly employ either deep neural networks trained offline on real‐world and/or synthetic data (Bechtel et al, ; Bojarski et al, ; C. Chen, Seff, Kornhauser, & Xiao, ; Eraqi et al, ; Fridman et al, ; Hecker et al, ; Rausch et al, ; Xu et al, ; S. Yang et al, ), or DRL systems trained and evaluated in simulation (Jaritz et al, ; Perot, Jaritz, Toromanoff, & Charette, ; Sallab et al, 2017b). Methods for porting simulation trained DRL models to real‐world driving have also been reported (Wayve, 2018), as well as DRL systems trained directly on real‐world image data (Pan et al, , ).…”

“…This kind of processing allows the training of large and complex network architectures, which in turn require huge amounts of training samples (see Section 8).End2End control papers mainly employ either deep neural networks trained offline on real-world and/or synthetic data(Bechtel et al, 2018;Bojarski et al, 2016;C. Chen, Seff, Kornhauser, & Xiao, 2015;Eraqi et al, 2017;Fridman et al, 2017;Hecker et al, 2018;Rausch et al, 2017;Xu et al, 2017;S. Yang et al, 2017a), or DRL systems trained and evaluated in simulation(Jaritz et al, 2018;Perot, Jaritz, Toromanoff, & Charette, 2017;Sallab et al, 2017b).…”

A Survey of Deep Learning Applications to Autonomous Vehicle Control

“…Not only is this impractical if our goal is to continue in the left direction, but can result in unsafe behaviour where the DNN oscillates between left and right but never picking either direction. Aiming to provide autonomous vehicles with contextual awareness, Hecker et al [137] collected a data set with a 360-degree view from 8 cameras and a driver following a route plan. This data set was then used to train a DNN to predict steering wheel angle and velocities from example images and route plans in the data set.…”

A Survey on Visual Traffic Simulation: Models, Evaluations, and Applications in Autonomous Driving

“…Hecker et al . [HDVG18] learn a novel end‐to‐end driving model by integrating the information from surrounding 360‐degrees view cameras into the route planner. The network used in this approach directly maps the sensor outputs to low‐level driving maneuvers including steering angles and speed.…”

“…Drive360 [HDVG18] includes 60 h of driving video from eight surround-view cameras. Low-level driving maneuvers (e.g.…”