Attention for Vision-Based Assistive and Automated Driving: A Review of Algorithms and Datasets

Abstract

Driving safety has been a concern since the first cars appeared on the streets. Driver inattention has been singled out as a major cause of accidents early on. This is hardly surprising, as drivers routinely perform other tasks in addition to controlling the vehicle. Decades of research into what causes lapses or misdirection of drivers' attention resulted in improvements in road safety through better design of infrastructure, driver training programs, in-vehicle interfaces, and, more recently, the development of driving assistance systems (ADAS) and driving automation. This review focuses on the methods for modeling and detecting spatio-temporal aspects of drivers' attention, i.e. where and when they look, for the two latter categories of applications.We start with a brief theoretical background on human visual attention, methods for recording and measuring attention in the driving context, types of driver inattention, and factors causing it. We then discuss machine learning approaches for 1) modeling gaze for assistive and self-driving applications and 2) detecting gaze for driver monitoring. Following the overview of state-of-the-art models, we provide an extensive list of publicly available datasets that feature recordings of drivers' gaze and other attention-related annotations. We conclude with a general overview of the remaining challenges, such as data availability and quality, evaluation methods, and the limited scope of attention modeling, and outline steps toward rectifying some of these issues. Categorized and annotated lists of the reviewed models and datasets are available at https://github.com/ykotseruba/attention_and_driving

Do They Want to Cross? Understanding Pedestrian Intention for Behavior Prediction

“…Models for predicting drivers' gaze can likewise be subdivided by the types of attentional mechanisms they model and stimuli they use. The majority of deep learning methods achieve promising results on drivers' gaze prediction benchmarks by learning associations between images of the scene and human saliency maps in a bottomup fashion [26]. Image-based models use CNNs pretrained on an image classification task to encode visual information and then apply convolution and upsampling layers to generate gaze maps [27], [28].…”

Computer Vision Applications in Intelligent Transportation Systems: A Survey

“…A study of CV applications designed to improve safety, operational efficiency, security, and the enforcement of laws in road transportation systems was presented in [ 7 ]. In [ 382 ], the authors examined ML methods and publicly available datasets that model the direction of a driver’s gaze by analyzing the driver’s spatiotemporal viewpoints for driving assistance and automation applications. Moreover, the authors provided a summary of current challenges and open issues, such as the availability and quality of data, evaluation techniques, and the limited scope of attention modeling, that need to be solved to make attention-based driving assistive systems applicable in automated systems.…”

Driver Attention Prediction based on Multi-ScaleFeature Fusion

“…These ADAS technologies have demonstrated their effectiveness in mitigating the risk of traffic accidents [5] [6]. While the current ADAS possess the capability to detect specific hazardous situations and autonomously initiate actions to avoid potential collisions, they lack integration with the driver's state or intention [7]. Consequently, there has been a growing research focus on exploring human perception mechanisms within visual perception systems for assisted driving vehicles, with driver attention emerging as a crucial aspect [8].…”