HARMONY: A Human-Centered Multimodal Driving Study in the Wild

Abstract

Effective shared autonomy requires a clear understanding of driver's behavior, which is governed by multiple psychophysiological and environmental variables. Disentangling this intricate web of interactions requires understanding the driver's state and behaviors in different real-world scenarios, longitudinally. Naturalistic Driving Studies (NDS) have shown to be an effective approach to understanding the driver's state and behavior in real-world scenarios. However, due to the lack of technological and computing capabilities, former NDS only focused on vision-based approaches, ignoring important psychophysiological factors such as cognition and emotion. The main objective of this paper is to introduce HARMONY, a human-centered multimodal naturalistic driving study, where driver's behaviors and states are monitored through (1) in-cabin and outside video streams (2) physiological signals including driver's heart rate and hand acceleration (IMU data), (3) ambient noise, light, and the vehicle's GPS location, and (4) music logs, including song features such as tempo. HARMONY is the first study that collects long-term naturalistic facial, physiological, and environmental data simultaneously. This paper summarizes HARMONY's goals, framework design, data collection and analysis, and the on-going and future research efforts. Through a presented case study, we first demonstrate the importance of longitudinal driver state sensing through using Kernel Density Estimation Methods. Second, we leverage the application of Bayesian Change Point detection methods to demonstrate how we can identify driver behaviors and responses to the environmental conditions by fusing psychophysiological information with features extracted from video streams.INDEX TERMS Naturalistic driving study, physiological sensing,driver state detection, shared-autonomy, contextual awareness, human-in-the-loop systems

ORCLSim: A System Architecture for Studying Bicyclist and Pedestrian Physiological Behavior Through Immersive Virtual Environments

“…We use bcp package in R (Erdman and Emerson 2007) to implement the change point analysis. A similar approach has been utilized in a previous study to identify changes in driver's HR data in different roadway conditions (Tavakoli et al 2021b). The BCP output is a time series data of probability of change points.…”

Multimodal Driver State Modeling through Unsupervised Learning

“…Analysis of historical driving data, which is often retrieved through naturalistic driving studies (NDS), also comes with problems such as the difficulty (time and cost) in analyzing the massive amount of collected information [9]. NDS is often conducted in a longitudinal fashion to help detect behaviors while different environmental noise and challenges exist in the data [10], which drastically increases the amount of data that needs to be analyzed for detecting specific driver behaviors, actions, and responses. One method to address this issue is to apply unsupervised learning on both driving behaviors as well as the driver's state.…”

“…Previous NDS data often lack information related to driver's physiological responses and cognitive metrics [10], [11]. With the current advancements in wearable technology, it is now viable to detect driver's psychophysiological states through monitoring their heart rate (HR), skin temperature, skin conductance, arm movement, and other physiological metrics.…”

“…With the current advancements in wearable technology, it is now viable to detect driver's psychophysiological states through monitoring their heart rate (HR), skin temperature, skin conductance, arm movement, and other physiological metrics. In line with these improvements, recent studies have used driver's physiological responses such as changes in HR to detect driver's stress levels correlated with the increase in HR values [10], [12]. Additionally, with the improvements in computer vision applications, it is now viable to use raw videos for estimating driver's breathing patterns, HR, gaze directions, and emotional features through facial landmarks [13].…”

“…Additionally, with the improvements in computer vision applications, it is now viable to use raw videos for estimating driver's breathing patterns, HR, gaze directions, and emotional features through facial landmarks [13]. As a result of these improvements in technology, recent NDS are starting to implement novel sensors and technologies to collect such data [10], [12], [14]- [16]. Utilizing these data streams and by applying unsupervised machine learning techniques, we can understand a driver's response within each driving behavior.…”