CogEmoNet: A Cognitive-Feature-Augmented Driver Emotion Recognition Model for Smart Cockpit

Li, Wenbo; Zeng, Guanzhong; Zhang, Juncheng; Xu, Yan; Xing, Yang; Zhou, Rui; Guo, Gang; Shen, Yu; Cao, Dongpu; Wang, Fei‐Yue

doi:10.1109/tcss.2021.3127935

Cited by 54 publications

(17 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The emotions of human drivers need to be induced by appropriate stimuli to collect emotion data. Video-audio clips have been proven to reliably trigger the emotions of human driver 6 , 34 , 35 . Based on the results of the questionnaire survey, we manually selected the seven most effective (the highest percentage of participants were selected in each emotional scenario) video-audio clips on the Bilibili website ( https://www.bilibili.com/ ) to induce the corresponding emotions of the human driver in Experiment III.…”

Section: Methodsmentioning

confidence: 99%

“…Moreover, by applying various machine learning techniques, based on the collected driving behaviour, EEG, facial expressions, driving posture, and road scene information, the dataset can be used to develop single-modal/multi-modal driver emotion monitoring algorithms 35 , 50 . Accurate and efficient emotion monitoring algorithms will help the emotion-aware interaction between human drivers and intelligent vehicles to improve driving safety and comfort, and increase human trust in machines 46 , 51 .…”

Section: Usage Notesmentioning

confidence: 99%

See 1 more Smart Citation

A multimodal psychological, physiological and behavioural dataset for human emotions in driving tasks

Tan

Xing

et al. 2022

Sci Data

Self Cite

View full text Add to dashboard Cite

Human emotions are integral to daily tasks, and driving is now a typical daily task. Creating a multi-modal human emotion dataset in driving tasks is an essential step in human emotion studies. we conducted three experiments to collect multimodal psychological, physiological and behavioural dataset for human emotions (PPB-Emo). In Experiment I, 27 participants were recruited, the in-depth interview method was employed to explore the driver’s viewpoints on driving scenarios that induce different emotions. For Experiment II, 409 participants were recruited, a questionnaire survey was conducted to obtain driving scenarios information that induces human drivers to produce specific emotions, and the results were used as the basis for selecting video-audio stimulus materials. In Experiment III, 40 participants were recruited, and the psychological data and physiological data, as well as their behavioural data were collected of all participants in 280 times driving tasks. The PPB-Emo dataset will largely support the analysis of human emotion in driving tasks. Moreover, The PPB-Emo dataset will also benefit human emotion research in other daily tasks.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Usage Notesmentioning

confidence: 99%

A multimodal psychological, physiological and behavioural dataset for human emotions in driving tasks

Tan

Xing

et al. 2022

Sci Data

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the intelligent cockpit, the perception, cognition, and decision of the driver's anger, as well as the regulation of the driver's anger through different interaction methods are an effective way to reduce the driving risk caused by anger and thereby improve road traffic safety. Benefiting from the rapid development of technologies, such as artificial intelligence, existing research has made great progress in driver anger detection and different interaction technologies development [34], [35], [36]. However, few studies have focused on what decisions the intelligent cockpit should make after recognizing anger, and generating corresponding strategies to regulate driver anger effectively.…”

Section: ) Intelligent Cockpit Perceptionmentioning

confidence: 99%

“…1) Anger Induction Material: Previous research has found that video clips are an effective method for driver emotion induction [33], [36], [58]. Therefore, selected video clips were played to stimulate the participants' anger.…”

Section: B Materials and Apparatusmentioning

confidence: 99%

Intelligent Cockpit for Intelligent Vehicle in Metaverse: A Case Study of Empathetic Auditory Regulation of Human Emotion

Xue

et al. 2023

IEEE Trans. Syst. Man Cybern, Syst.

Self Cite

View full text Add to dashboard Cite

Advances in technologies, such as intelligent connected vehicles and the metaverse are driving the rapid development of automotive intelligent cockpits. From the perspective of the cyber-physical-social system (CPSS), this study proposed the intelligent cockpit composition framework which includes three layers of perception, cognition and decision, and interaction. Meanwhile, we also describe the relationship between the intelligent cockpit framework and the outside environment. The framework can dynamically perceive and understand humans, and provide feedback on the understanding results, which is beneficial to provide a safe, efficient, and enjoyable experience for humans in the intelligent cockpit. In the cognition and decision layers of the proposed framework, we design a case study of active empathetic auditory regulation of driver anger, focusing on improving road traffic safety. We conducted an in-depth interview experiment and designed two auditory regulation materials of active empathy speech and text-to-speech (TTS) speech. Next, 30 participants were recruited, and they completed a total of 240 anger-regulated driving experiments in the straight and obstacle avoidance scenarios. Finally, we quantitatively analyzed and compared the participants' subjective feelings, physiological changes, driving behaviors, and driving risks, as well as validated the driver anger regulation quality of AES and TTS. The proposed research methods results are beneficial to the design of future intelligent cockpit emotion regulation systems, toward a better intelligent cockpit.

show abstract

“…Wenbo et al [54] investigated the driver anger regulation in visual attributes. Driver facial expression mirrors driver emotional state [55]; therefore, it is important for emotion recognition and ensuring driving safety.…”

Section: Driver Facial Expression Recognitionmentioning

confidence: 99%

On-Road Driver Emotion Recognition Using Facial Expression

Xiao

Zeng

et al. 2022

Applied Sciences

Self Cite

View full text Add to dashboard Cite

With the development of intelligent automotive human-machine systems, driver emotion detection and recognition has become an emerging research topic. Facial expression-based emotion recognition approaches have achieved outstanding results on laboratory-controlled data. However, these studies cannot represent the environment of real driving situations. In order to address this, this paper proposes a facial expression-based on-road driver emotion recognition network called FERDERnet. This method divides the on-road driver facial expression recognition task into three modules: a face detection module that detects the driver’s face, an augmentation-based resampling module that performs data augmentation and resampling, and an emotion recognition module that adopts a deep convolutional neural network pre-trained on FER and CK+ datasets and then fine-tuned as a backbone for driver emotion recognition. This method adopts five different backbone networks as well as an ensemble method. Furthermore, to evaluate the proposed method, this paper collected an on-road driver facial expression dataset, which contains various road scenarios and the corresponding driver’s facial expression during the driving task. Experiments were performed on the on-road driver facial expression dataset that this paper collected. Based on efficiency and accuracy, the proposed FERDERnet with Xception backbone was effective in identifying on-road driver facial expressions and obtained superior performance compared to the baseline networks and some state-of-the-art networks.

show abstract

CogEmoNet: A Cognitive-Feature-Augmented Driver Emotion Recognition Model for Smart Cockpit

Cited by 54 publications

References 46 publications

A multimodal psychological, physiological and behavioural dataset for human emotions in driving tasks

A multimodal psychological, physiological and behavioural dataset for human emotions in driving tasks

Intelligent Cockpit for Intelligent Vehicle in Metaverse: A Case Study of Empathetic Auditory Regulation of Human Emotion

On-Road Driver Emotion Recognition Using Facial Expression

Contact Info

Product

Resources

About