Multimodal Data Collection System for Driver Emotion Recognition Based on Self-Reporting in Real-World Driving

Abstract

As vehicles provide various services to drivers, research on driver emotion recognition has been expanding. However, current driver emotion datasets are limited by inconsistencies in collected data and inferred emotional state annotations by others. To overcome this limitation, we propose a data collection system that collects multimodal datasets during real-world driving. The proposed system includes a self-reportable HMI application into which a driver directly inputs their current emotion state. Data collection was completed without any accidents for over 122 h of real-world driving using the system, which also considers the minimization of behavioral and cognitive disturbances. To demonstrate the validity of our collected dataset, we also provide case studies for statistical analysis, driver face detection, and personalized driver emotion recognition. The proposed data collection system enables the construction of reliable large-scale datasets on real-world driving and facilitates research on driver emotion recognition. The proposed system is avaliable on GitHub.

MDEmoNet: A Multimodal Driver Emotion Recognition Network for Smart Cockpit

“…Li et al [22] publicly released the first and currently the only multimodal dataset for driving tasks, named PPB-Emo, in 2022. Additionally, Oh et al [23] proposed an onboard system for collecting multimodal data in real driving environments. In recent years, research on driver emotion recognition has been trending towards the direction of multimodality.…”

Special Issue “Emotion Intelligence Based on Smart Sensing”

“…Finally, this issue also extends to studies applicable to the real world (e.g., in driving, games, and virtual agents). The authors of [ 12 ] proposed a data collection system that collects multimodal emotion datasets during real-world driving. The proposed system includes a self-reportable HMI application into which a driver directly inputs their current emotion state.…”