Machine Learning and End-to-End Deep Learning for Monitoring Driver Distractions From Physiological and Visual Signals

Abstract

It is only a matter of time until autonomous vehicles become ubiquitous; however, human driving supervision will remain a necessity for decades. To assess the driver's ability to take control over the vehicle in critical scenarios, driver distractions can be monitored using wearable sensors or sensors that are embedded in the vehicle, such as video cameras. The types of driving distractions that can be sensed with various sensors is an open research question that this study attempts to answer. This study compared data from physiological sensors (palm electrodermal activity (pEDA), heart rate and breathing rate) and visual sensors (eye tracking, pupil diameter, nasal EDA (nEDA), emotional activation and facial action units (AUs)) for the detection of four types of distractions. The dataset was collected in a previous driving simulation study. The statistical tests showed that the most informative feature/modality for detecting driver distraction depends on the type of distraction, with emotional activation and AUs being the most promising. The experimental comparison of seven classical machine learning (ML) and seven end-to-end deep learning (DL) methods, which were evaluated on a separate test set of 10 subjects, showed that when classifying windows into distracted or not distracted, the highest F1-score of 79% was realized by the extreme gradient boosting (XGB) classifier using 60-second windows of AUs as input. When classifying complete driving sessions, XGB's F1-score was 94%. The best-performing DL model was a spectro-temporal ResNet, which realized an F1-score of 75% when classifying segments and an F1-score of 87% when classifying complete driving sessions. Finally, this study identified and discussed problems, such as label jitter, scenario overfitting and unsatisfactory generalization performance, that may adversely affect related ML approaches.

Can We Ditch Feature Engineering? End-to-End Deep Learning for Affect Recognition from Physiological Sensor Data

“…DL architectures for signal processing have not yet realized any outstanding breakthrough, and designing them remains challenging, especially for problems with limited data for training. The main aim of the experiments was to compare ten end-to-end DL architectures (see Section 3.1 ): Fully convolutional network (FCN) [ 44 ], Residual network (Resnet) [ 45 ], Multi layer perceptron (MLP) [ 44 ], Encoder [ 46 ], Time convolutional neural network (Time-CNN) [ 47 ], Multichannel deep convolutional neural network (MCDCNN) [ 47 ], Spectrotemporal residual network (Stresnet) [ 48 ], Convolutional neural network with long-short term memory (CNN-LSTM) [ 5 ], Multi layer perceptron with long-short term memory (MLP-LSTM), and InceptionTime [ 49 ]. …”

Active Vision-Based Attention Monitoring System for Non-Distracted Driving

“…For example, Ramírez et al [34] and Takemura et al [35] proposed a method that uses a head mounted sensors. A few researches have been conducted recently for driver's attention detection by integrating various approaches, for example, combining driver's physiological signals and visual signal [36], driver's physiological signals and driving contexts [37] or driver's visual cues and driving patterns [38].…”

Towards Robust and Interpretable Practical Applications of Automatic Mental State Analysis Using a Dynamic and Hybrid Facial Action Estimation Approach

“…The results of this study are presented in Preprint C.1.1 (Update 02.08.2020: This is now published in IEEE Access. See [56]).…”

“…4 This study is described in detail in Preprint C.1.1 (Update 02.08.2020: This is now published in IEEE Access. See [56]). It was found that the intensities of AU25-LipsPart produced the most informative feature for recognising cognitive distraction.…”

“…This inspired an effort to automatically detect driver distraction using the AU intensity estimates (see Preprint C.1.1. Update 02.08.2020: See publication [56]).…”