Emotion recognition and adaptation in spoken dialogue systems

Abstract

The involvement of emotional states in intelligent spoken human-computer interfaces has evolved to a recent field of research. In this article we describe the enhancements and optimizations of a speech-based emotion recognizer jointly operating with automatic speech recognition. We argue that the knowledge about the textual content of an utterance can improve the recognition of the emotional content. Having outlined the experimental setup we present results and demonstrate the capability of a postprocessing algorithm combining multiple speech-emotion recognizers. For the dialogue management we propose a stochastic approach comprising a dialogue model and an emotional model interfering with each other in a combined dialogue-emotion model. These models are trained from dialogue corpora and being assigned different weighting factors they determine the course of the dialogue.

Speaker-independent emotion recognition exploiting a psychologically-inspired binary cascade classification schema

“…An emotion recognizer that may operate jointly with an automatic speech recognizer is examined by Pitterman et al [46]. The feature vector comprises of MFCCs (along with their first-and second-order differences), intensity, and three formants, along with pitch and pitch statistics, namely minimum, mean, maximum, deviation and range.…”

“…No feature selection technique is applied, while the HMMs are employed as classifiers to a speaker-dependent protocol, contrary to our approach that applies feature selection and a speaker-independent protocol. Also, speech recognition is not a prerequisite in this work, whereas the stated set of features in [46] is a subset of the feature vector computed by the authors. However, in both cases the authors compute the first-and second-differences of the features in order to capture their temporal evolution.…”

“…In [13], it is stated that emotional signals are chaotic in nature whereas it is a fact that researchers have not yet found a sufficient feature set to describe the emotional states efficiently [20]. It is also proven that each speaker expresses his/her emotions in a different manner [20] and that the two genders convey their emotions in profoundly different ways [20] [40] [46] [60] [66]. Furthermore, the same linguistic content may bear different emotional state.…”

Modeling the user state for context-aware spoken interaction in ambient assisted living

“…The dialog model proposed by [121] combined three different submodels: an emotional model describing the transitions between user emotional states during the interaction regardless of the data content, a plain dialog model describing the transitions between existing dialog states regardless of the emotions, and a combined model including the dependencies between combined dialog and emotional states. Then, the next dialog state was derived from a combination of the plain dialog model and the combined model.…”

“…In our proposal, we employ statistical techniques for inferring user acts, which makes it easier to port it to different application domains. Also the proposed architecture is modular and thus makes it possible to employ different emotion and intention recognizers, as the intention recognizer is not linked to the dialog manager as in [121].…”