Deep Learning-Based Drivers Emotion Classification System in Time Series Data for Remote Applications

Abstract

Aggressive driving emotions is indeed one of the major causes for traffic accidents throughout the world. Real-time classification in time series data of abnormal and normal driving is a keystone to avoiding road accidents. Existing work on driving behaviors in time series data have some limitations and discomforts for the users that need to be addressed. We proposed a multimodal based method to remotely detect driver aggressiveness in order to deal these issues. The proposed method is based on change in gaze and facial emotions of drivers while driving using near-infrared (NIR) camera sensors and an illuminator installed in vehicle. Driver's aggressive and normal time series data are collected while playing car racing and truck driving computer games, respectively, while using driving game simulator. Dlib program is used to obtain driver's image data to extract face, left and right eye images for finding change in gaze based on convolutional neural network (CNN). Similarly, facial emotions that are based on CNN are also obtained through lips, left and right eye images extracted from Dlib program. Finally, the score level fusion is applied to scores that were obtained from change in gaze and facial emotions to classify aggressive and normal driving. The proposed method accuracy is measured through experiments while using a self-constructed large-scale testing database that shows the classification accuracy of the driver's change in gaze and facial emotions for aggressive and normal driving is high, and the performance is superior to that of previous methods.Association Foundation for Traffic safety, published in 2009, that the aggressive behavior of driver causes 56% of traffic accidents [2]. Besides precious human lives, people, company, and government also lose billions of dollars due to road accidents. For this reason, aggressive driving behavior must be strongly discouraged that will result in reduction of the number of traffic accidents.The classification of aggressive and normal behavior is an important issue that can be used to increase awareness of driving habits of drivers as many drivers are over confident and are unaware of their bad driving habits [3]. If we can automatically identify the drivers driving behaviors, the drivers can be aware of their bad habits and assist them to avoid potential car accidents. Other than this if, monitoring results could be sent back to a security observing server of the local police station that could help to automatically detect aggressive drivers. The conventional method to keep a check on aggressive driving is by police patrolling, but, due to lack of police force, all roads cannot be simultaneously monitored and it also costs a lot [4]. The need of intelligent surveillance system is increasing with the increase in population. The advance driver assistance system (ADAS) that can monitor driver's attention and driving behavior can improve road safety, which will also enhance the effectiveness of the ADAS [5]. Many challenges are faced by these real time systems that ...

Deep-Net: A Lightweight CNN-Based Speech Emotion Recognition System Using Deep Frequency Features

“…All the CNNs models are extremely good working in the spatial domain to extract the salient features from the data [ 15 ]. Recently, fully convolutional networks (FCNs) have been introduced as a variant of the CNNs, which handle inputs of variable sizes based on their properties, and they have achieved state-of-the-art accuracy in time-series problems [ 16 ]. However, the FCNs model is not able to learn temporal features in this regard.…”

Object Detection, Distributed Cloud Computing and Parallelization Techniques for Autonomous Driving Systems

“…These NNs are trained and tested with an extensive data set to detect any change in gaze and facial emotions, thus discerning regular and aggressive driving [70] or distracted driving actions [71]. In some other cases, there want to be able to predict drivers' behavior; to make this possible CRNNs are used.…”

Real-Time Abnormal Event Detection for Enhanced Security in Autonomous Shuttles Mobility Infrastructures

“…Finally, once these images are fed as inputs into a 3D convolutional neural network and a 2D convolutional neural network, respectively, the model is able to perform action recognition. Other approaches focused on the transport domain, such as [ 21 , 22 , 23 , 24 ], rely on facial expressions and behavior, gaze and eye tracking for detecting emotions, stress, anxiety, and panic. Although effective, those methods add another layer of complexity to our solution, as they require high resolution facial images which subsequently introduce privacy issues.…”