Driver Drowsiness Estimation Based on Factorized Bilinear Feature Fusion and a Long-Short-Term Recurrent Convolutional Network

Abstract

The effective detection of driver drowsiness is an important measure to prevent traffic accidents. Most existing drowsiness detection methods only use a single facial feature to identify fatigue status, ignoring the complex correlation between fatigue features and the time information of fatigue features, and this reduces the recognition accuracy. To solve these problems, we propose a driver sleepiness estimation model based on factorized bilinear feature fusion and a long- short-term recurrent convolutional network to detect driver drowsiness efficiently and accurately. The proposed framework includes three models: fatigue feature extraction, fatigue feature fusion, and driver drowsiness detection. First, we used a convolutional neural network (CNN) to effectively extract the deep representation of eye and mouth-related fatigue features from the face area detected in each video frame. Then, based on the factorized bilinear feature fusion model, we performed a nonlinear fusion of the deep feature representations of the eyes and mouth. Finally, we input a series of fused frame-level features into a long-short-term memory (LSTM) unit to obtain the time information of the features and used the softmax classifier to detect sleepiness. The proposed framework was evaluated with the National Tsing Hua University drowsy driver detection (NTHU-DDD) video dataset. The experimental results showed that this method had better stability and robustness compared with other methods.

A Framework for Instantaneous Driver Drowsiness Detection Based on Improved HOG Features and Naïve Bayesian Classification

“…As it might be expected, the achieved results demonstrate that the presented system can operate reliably and efficiently, achieving real-time performance operation for video sequences, due to the use of highly efficient algorithmic implementations in OpenCV library in combination with custom C++ functions. [41] 75.80 Scale-Pruned 3D-CNN [39] 78.48 seqMT-DMF [37] 83.44 MSTN [38] 82.61 Joint-Shape RF [42] 88.18 3D-DCNN [18] 87.46 Human [17] 80.83…”

Multi-Level Classification of Driver Drowsiness by Simultaneous Analysis of ECG and Respiration Signals Using Deep Neural Networks

“…One of the best solutions to compensate for these shortcomings is to simultaneously use sensors from other sources. Previous studies have shown that vehicle-based and facial measurements are rather accurate in multi-level drowsiness classifications [ 44 , 61 , 62 ], but they can be affected by the road geometry and lighting conditions. Fusing physiological features with vehicle dynamics and the drivers’ facial features improve the classification accuracy and compensate for the temporary loss of signals.…”

Advanced e-Call Support Based on Non-Intrusive Driver Condition Monitoring for Connected and Autonomous Vehicles

“…For reducing the motion artifacts in the use of non-intrusive ECG sensors, the authors of [25] propose an interesting approach: the use of two capacitive ECG sensors (cECGs) for determining the ECG, with an additional two cECGs to obtain the information on motion. Fatigue feature extraction, fatigue feature fusion and driver drowsiness detection are used in [26] in a particular model employing convolutional neural networks (CNNs), applied to video streaming from a camera pointed to the driver. Not directly connected with driving cars and detecting sleepiness in the automotive field, but closely related to it, is a study [26] concerning the use of special light glasses in reducing sleepiness.…”

“…Fatigue feature extraction, fatigue feature fusion and driver drowsiness detection are used in [26] in a particular model employing convolutional neural networks (CNNs), applied to video streaming from a camera pointed to the driver. Not directly connected with driving cars and detecting sleepiness in the automotive field, but closely related to it, is a study [26] concerning the use of special light glasses in reducing sleepiness. Causes and effects that lead to drowsiness state are analyzed in [27], and the result of this study classifies the first causes of drowsiness as being: exposure to recent stress situations, medication and sleep deprivation.…”