Impact of cognitive load and frustration on drivers’ speech.

Abstract

Secondary tasks such as cell phone calls or interaction with automated speech dialog systems (SDSs) increase the driver’s cognitive load as well as the probability of driving errors. This study analyzes speech production variations due to cognitive load and emotional state of drivers in real driving conditions. Speech samples were acquired from 24 female and 17 male subjects (approximately 8.5 h of data) while talking to a co-driver and communicating with two automated call centers, with emotional states (neutral, negative) and the number of necessary SDS query repetitions also labeled. A consistent shift in a number of speech production parameters (pitch, first format center frequency, spectral center of gravity, spectral energy spread, and duration of voiced segments) was observed when comparing SDS interaction against co-driver interaction; further increases were observed when considering negative emotion segments and the number of requested SDS query repetitions. A mel frequency cepstral coefficient based Gaussian mixture classifier trained on 10 male and 10 female sessions provided 91% accuracy in the open test set task of distinguishing co-driver interactions from SDS interactions, suggesting—together with the acoustic analysis—that it is possible to monitor the level of driver distraction directly from their speech.

Multi-modal highlight generation for sports videos using an information-theoretic excitability measure

“…Past literature on emotions and stress suggests that a number of speech production parameters can be affected by varying speech modalities [18][19][20][21]. In this section, we extract a set of speech parameters derived from the linear model of speech production and evaluate their correlation with the perceptual excitement level in the commentators audio.…”

“…We have observed that SES, when combined with SCG, constitutes a more noise-robust spectral descriptor for emotion and stress classification than spectral slope [20]. The distribution of game-specific means of F 0 , F 1 , F 2 , and SCG across the four perceptual excitement levels is summarized in Figures 2, 3 degree of correlation between the speech parameters and perceived excitement levels, a linear regression was conducted.…”

“…Past studies [5,6] typically consider simplistic features such as energy, zero crossing rate, and others to estimate excitement from the audio modality. Inspired by studies on emotion assessment [18][19][20], in this study, we analyze the effects of excitement on the parameters of a linear model of speech production derived from commentators' speech. As will be shown, some speech parameters are strongly correlated with the perceptual excitability and hence, are selected to form a feature vector for the audio-based excitement assessment.…”

Predicting user mental states in spoken dialogue systems

“…Boril et al [36] measured speech production variations during the interactions of drivers with commercial automated dialogue systems. They discussed that cognitive load and emotional states affect the number of query repetitions required for the users to obtain the information they are looking for.…”

A New Fuzzy Cognitive Map Learning Algorithm for Speech Emotion Recognition

“…As an important link in the human-computer interaction system, this method has received increasing attention in recent years. Speech emotion recognition has been widely applied to medical diagnosis [1], drowsy driving detection [2], and emotional state monitoring of students during the e-learning process [3].…”