DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving

Abstract

Today, there are two major paradigms for vision-based autonomous driving systems: mediated perception approaches that parse an entire scene to make a driving decision, and behavior reflex approaches that directly map an input image to a driving action by a regressor. In this paper, we propose a third paradigm: a direct perception approach to estimate the affordance for driving. We propose to map an input image to a small number of key perception indicators that directly relate to the affordance of a road/traffic state for driving. Our representation provides a set of compact yet complete descriptions of the scene to enable a simple controller to drive autonomously. Falling in between the two extremes of mediated perception and behavior reflex, we argue that our direct perception representation provides the right level of abstraction. To demonstrate this, we train a deep Convolutional Neural Network using recording from 12 hours of human driving in a video game and show that our model can work well to drive a car in a very diverse set of virtual environments. We also train a model for car distance estimation on the KITTI dataset. Results show that our direct perception approach can generalize well to real driving images. Source code and data are available on our project website.

Safe Visual Navigation via Deep Learning and Novelty Detection

“…Barnes et al [3] and Oliveira et al [24] provide more recent examples of deep learning used to segment image pixels into drivable routes, which can then be used for planning. While we focus on supervised learning of collision probability, there are many other ways to map between images and actions, including end-to-end neural networks [6,19], affordance-based representations [7], reinforcement learning using either simulated or real images [23,31], and classification of paths from manually collected data [11]. While it would be impossible to survey the vast literature on visual navigation here, we note that these examples and many others do not explicitly address uncertainty about the learned model.…”

Efficient Processing of Deep Neural Networks: A Tutorial and Survey

“…In 2015, the ImageNet winning entry, ResNet [15], exceeded human-level accuracy with a top-5 error rate 4 below 5%. Since then, the error rate has dropped below 3% and more focus is now being placed on more challenging components of the competition, such as object detection and localization.…”

“…• Robotics DNNs have been successful in the domain of robotic tasks such as grasping with a robotic arm [32], motion planning for ground robots [33], visual navigation [4,34], control to stabilize a quadcopter [35] and driving strategies for autonomous vehicles [36]. DNNs are already widely used in multimedia applications today (e.g., computer vision, speech recognition).…”

“…DNNs have significantly improved the accuracy of many computer vision tasks such as image classification [14], object localization and detection [18], image segmentation [19], and action recognition [20]. 4 The top-5 error rate is measured based on whether the correct answer appears in one of the top 5 categories selected by the algorithm.…”

“…Since the breakthrough application of DNNs to speech recognition [2] and image recognition [3], the number of applications that use DNNs has exploded. These DNNs are employed in a myriad of applications from self-driving cars [4], to detecting cancer [5] to playing complex games [6]. In many of these domains, DNNs are now able to exceed human accuracy.…”