Detecting unexpected obstacles for self-driving cars: Fusing deep learning and geometric modeling

Abstract

Abstract-The detection of small road hazards, such as lost cargo, is a vital capability for self-driving cars. We tackle this challenging and rarely addressed problem with a vision system that leverages appearance, contextual as well as geometric cues.To utilize the appearance and contextual cues, we propose a new deep learning-based obstacle detection framework. Here a variant of a fully convolutional network is used to predict a pixel-wise semantic labeling of (i) free-space, (ii) on-road unexpected obstacles, and (iii) background. The geometric cues are exploited using a state-of-the-art detection approach that predicts obstacles from stereo input images via model-based statistical hypothesis tests.We present a principled Bayesian framework to fuse the semantic and stereo-based detection results. The mid-level Stixel representation is used to describe obstacles in a flexible, compact and robust manner.We evaluate our new obstacle detection system on the Lost and Found dataset, which includes very challenging scenes with obstacles of only 5 cm height. Overall, we report a major improvement over the state-of-the-art, with relative performance gains of up to 50%. In particular, we achieve a detection rate of over 90% for distances of up to 50 m. Our system operates at 22 Hz on our self-driving platform.

A survey of deep learning techniques for autonomous driving

“…Deep learning is used in the context of occupancy maps either for dynamic objects detection and tracking (Ondruska, Dequaire, Wang, & Posner, ), probabilistic estimation of the occupancy map surrounding the vehicle (Hoermann, Bach, & Dietmayer, ; Ramos, Gehrig, Pinggera, Franke, & Rother, ), or for deriving the driving scene context (Marina et al, ; Seeger, Müller, & Schwarz, ). In the latter case, the OG is constructed by accumulating data over time, whereas a deep neural net is used to label the environment into driving context classes, such as highway driving, parking area, or inner‐city driving.…”

“…Deep learning is used in the context of occupancy maps either for dynamic objects detection and tracking (Ondruska, Dequaire, Wang, & Posner, 2016), probabilistic estimation of the occupancy map surrounding the vehicle (Hoermann, Bach, & Dietmayer, 2017;Ramos, Gehrig, Pinggera, Franke, & Rother, 2016), or for deriving the driving scene context Seeger, Müller, & Schwarz, 2016). In the latter case, the OG is constructed by accumulating data over time, whereas a deep neural net is used to F I G U R E 5 Semantic segmentation performance comparison on the CityScapes data set (Cityscapes, 2018 Occupancy maps represent an in-vehicle virtual environment, integrating perceptual information in a form better suited for path planning and motion control.…”

Securing Connected & Autonomous Vehicles: Challenges Posed by Adversarial Machine Learning and the Way Forward

“…In CAVs, accurate and timely prediction of different events encountered in driving scenes is another important task which is mainly accomplished using different ML and DL algorithms. For instance, autonomous vehicles uses DL models for the detection and localization of obstacles [76], different objects (e.g., vehicles, pedestrians, and bikes, etc.) [77] and their behavior (e.g., tracking pedestrians along the way [78]) and traffic signs [79] and traffic lights recognition [80].…”

On Assessing Driver Awareness of Situational Criticalities: Multi-modal Bio-Sensing and Vision-Based Analysis, Evaluations, and Insights

“…We calculated the PSD of three EEG bands namely theta (4-7 Hz), alpha (7-13 Hz) and Beta (13)(14)(15)(16)(17)(18)(19)(20)(21)(22)(23)(24)(25)(26)(27)(28)(29)(30) Hz) for all the EEG channels. The choice of these three specific EEG bands was made since they are the most commonly used bands and thought to carry a lot of information about human cognition.…”