Automatic Detection of a Driver’s Complex Mental States

Abstract

Automatic classification of drivers' mental states is an important yet relatively unexplored topic. In this paper, we define a taxonomy of a set of complex mental states that are relevant to driving, namely: Happy, Bothered, Concentrated and Confused. We present our video segmentation and annotation methodology of a spontaneous dataset of natural driving videos from 10 different drivers. We also present our real-time annotation tool used for labelling the dataset via an emotion perception experiment and discuss the challenges faced in obtaining the ground truth labels. Finally, we present a methodology for automatic classification of drivers' mental states. We compare SVM models trained on our dataset with an existing nearest neighbour model pre-trained on posed dataset, using facial Action Units as input features. We demonstrate that our temporal SVM approach yields better results. The dataset's extracted features and validated emotion labels, together with the annotation tool, will be made available to the research community.

DRER: Deep Learning–Based Driver’s Real Emotion Recognizer

“…Among the emotion recognition databases, some databases include facial image data [ 46 , 47 , 48 ] or bio-physiological signals [ 49 ] collected in driving conditions. The UTDrive DB Classic [ 46 ] collected in the context of the level of stress while 77 participants were driving in real-world urban areas highway conditions.…”

“…The data include video, audio, GPS location and car access. The database by Ma et al [ 47 ] classified the drivers’ emotional states into four categories (i.e., happy, bothered, concentrated and confused) and collected data while 10 drivers were driving around 24 km each. External annotators provided the emotional annotation, and the data were focused on the face video.…”

Emotional machines: The next revolution

“…In particular, this issue is undergoing a fast development in the automotive sector. Systems that monitor the attentive and emotional state of the driver [35] will be installed in cars, even before they will be fully autonomous.…”

Intelligent Driver Profiling System for Cars – A Basic Concept

“…Most of these systems use modern camera technology and image processing in order to evaluate possible distraction based on eye and head movement [30][31][32][33][34][35][36]. The next level of state monitoring does not only include the further improvement of general distraction or fatigue assessment but goes one step further and aims to detect complex mental states [37]. Such systems are crucial for a potential intelligent driver profiling system for cars since the information provided is required to enable car characteristic adjustments based on the driver's physical as well as mental conditions.…”