Action Units and Their Cross-Correlations for Prediction of Cognitive Load during Driving

Abstract

Abstract-Driving requires the constant coordination of many body systems and full attention of the person. Cognitive distraction (subsidiary mental load) of the driver is an important factor that decreases attention and responsiveness, which may result in human error and accidents. In this paper, we present a study of facial expressions of such mental diversion of attention. First, we introduce a multi-camera database of 46 people recorded while driving a simulator in two conditions, baseline and induced cognitive load using a secondary task. Then, we present an automatic system to differentiate between the two conditions, where we use features extracted from Facial Action Unit (AU) values and their cross-correlations in order to exploit recurring synchronization and causality patterns. Both the recording and detection system are suitable for integration in a vehicle and a real-world application, e.g. an early warning system. We show that when the system is trained individually on each subject we achieve a mean accuracy and F-score of ∼ 95%, and for the subject independent tests ∼ 68% accuracy and ∼ 66% F-score, with person-specific normalization to handle subject dependency. Based on the results, we discuss the universality of the facial expressions of such states and possible real-world uses of the system.

ADABase: A Multimodal Dataset for Cognitive Load Estimation

“…Using simple features based on the number of activated action units within a certain time frame, we found several action units that changed significantly for increasing levels of task demand of n -back and k -drive levels. These observations are supported by the findings of Yuce et al who analyzed the link between action units and cognitive load while driving [ 29 ]. We measured several performance metrics for all conducted n -back and k -drive tests and levels for both secondary and primary task loads, including precision and recall.…”

“…The Facial Action Coding System (FACS) system, introduced by Ekman and Friesen [ 99 ] and improved by Ekman and Rosenberg [ 100 ] specifies different action units. Martinez et al present the current state of research and its application [ 101 ], while predicting cognitive load using visual facial cues has been studied in Li and Busso [ 102 ], Yuce et al [ 29 ] and Viegas et al [ 28 ], on which we base our decision to extract the action units presented in Table 3 . For our statistical analysis and machine learning experiments, we count the frames with active actions unit within the given window and denote the number of action units similar to our physiological features with # as count.…”

“…A very prominent approach in the affective computing community is the extraction of action units from facial videos to detect emotions. These action units can also be useful to detect cognitive load while driving [ 28 , 29 ]. Another extensively researched modality is based on eye-tracking technology [ 30 , 31 ], that in addition to the behavioral dimension accounts to physiological measurements’ with parameters such as pupil diameter.…”

Cross-Task and Cross-Participant Classification of Cognitive Load in an Emergency Simulation Game

“…For instance, (truck) drivers or other workers controlling complex engines may be prompted to take breaks or can be provided with individualized training when detected to be over-strained in specific situations (e.g. [6] or [7]). Other examples might be conceivable in the medical domain where surgeons might be relieved when necessary, or in aviation scenarios where pilots may be provided with support from their copilots or from assistance systems depending on their cognitive-load levels [8].…”