A Multimodal Approach for Monitoring Driving Behavior and Emotions

Abstract

Founded in 1991, the Mineta Transportation Institute (MTI), an organized research and training unit in partnership with the Lucas College and Graduate School of Business at San José State University (SJSU), increases mobility for all by improving the safety, efficiency, accessibility, and convenience of our nation's transportation system. Through research, education, workforce development, and technology transfer, we help create a connected world. MTI leads the four-university. MTI leads the four-university California State University Transportation Consortium funded by the State of California through Senate Bill 1. MTI's transportation policy work is centered on three primary responsibilities: MINETA TRANSPORTATION INSTITUTE Research MTI works to provide policy-oriented research for all levels of government and the private sector to foster the development of optimum surface transportation systems. Research areas include: bicycle and pedestrian issues; financing public and private sector transportation improvements; intermodal connectivity and integration; safety and security of transportation systems; sustainability of transportation systems; transportation / land use / environment; and transportation planning and policy development. Certified Research Associates conduct the research. Certification requires an advanced degree, generally a Ph.D., a record of academic publications, and professional references. Research projects culminate in a peer-reviewed publication, available on TransWeb, the MTI website (http://transweb.sjsu.edu). Education The Institute supports education programs for students seeking a career in the development and operation of surface transportation systems. MTI, through San José State University, offers an AACSBaccredited Master of Science in Transportation Management and graduate certificates in Transportation Management, Transportation Security, and High-Speed Rail Management that serve to prepare the nation's transportation managers for the 21st century. With the active assistance of the California Department of Transportation (Caltrans), MTI delivers its classes over a state-of-the-art videoconference network throughout the state of California and via webcasting beyond, allowing working transportation professionals to pursue an advanced degree regardless of their location. To meet the needs of employers seeking a diverse workforce, MTI's education program promotes enrollment to under-represented groups. Information and Technology Transfer MTI utilizes a diverse array of dissemination methods and media to ensure research results reach those responsible for managing change. These methods include publication, seminars, workshops, websites, social media, webinars, and other technology transfer mechanisms. Additionally, MTI promotes the availability of completed research to professional organizations and journals and works to

Persona-PhysioSync AV: Personalized Interaction through Personality and Physiology Monitoring in Autonomous Vehicles

“…We suggest a framework that will be based on the development of a continuous monitoring protocol of passengers' comfort [55], engagement [56,57], stress [58,59], and fatigue levels [60] through multimodal physiological and emotional parameters [61] adjusted to their personality traits (Figure 1) [62]. By proposing this protocol, we aim to create a new way for passengers and drivers to interact with their AVs.…”

“…Feasibility has been demonstrated in measuring individual responses using physiological indices, including stress and emotions while driving, in real-life scenarios [69]. Further evidence of the possibility of evaluating users' emotional responses using physiology was shown in [61,[70][71][72] using relevant simulators [73,74]. Tavakoli et al (2021) presented the HARMONY experimental platform, which can analyze drivers' states in naturalistic driving studies using multimodal data collection [71].…”

Comprehensive Assessment of Artificial Intelligence Tools for Driver Monitoring and Analyzing Safety Critical Events in Vehicles

“…Also, some studies focused on the relationship between drivers' emotions and driving circumstances. For instance, Balali et al [11] had a naturalistic driving setup consisting of videos recording both the driver and the road via a Z-Edge S3 Dual Dashcam, heart rate data, and data from the car's Controller Area Network. Results suggested that weather conditions and road types may significantly change driver emotions and driving behavior.…”

“…The physiological approach attaches sensors to drivers to obtain human body signals (heart rate, skin conductance, respiration rate, skin temperature, etc.) for DMSs [8,11,12]. This approach is used to detect certain driver states, such as fatigue or stress; however, drivers often consider the attached sensors to be intrusive.…”

“…[10,16]. In what has traditionally been a field of simple metrics and labor-intensive video review, the transportation industry and related academia have explored AI approaches for improved DMSs by combining two or three of these approaches via DL and data fusion [11,12,17]. The AI methods provide comprehensive datasets to better understand how drivers react under different environmental factors, including both the road scene and the psychological state of the driver, to provide recommendations for optimizing DMS output towards the identification of at-risk driver behaviors and conditions using suitable system devices and signals.…”