Acoustic and lexical representations for affect prediction in spontaneous conversations

Abstract

In this article we investigate what representations of acoustics and word usage are most suitable for predicting dimensions of affect|AROUSAL, VALANCE, POWER and EXPECTANCY|in spontaneous interactions. Our experiments are based on the AVEC 2012 challenge dataset. For lexical representations, we compare corpus-independent features based on psychological word norms of emotional dimensions, as well as corpus-dependent representations. We find that corpus-dependent bag of words approach with mutual information between word and emotion dimensions is by far the best representation. For the analysis of acoustics, we zero in on the question of granularity. We confirm on our corpus that utterance-level features are more predictive than word-level features. Further, we study more detailed representations in which the utterance is divided into regions of interest (ROI), each with separate representation. We introduce two ROI representations, which significantly outperform less informed approaches. In addition we show that acoustic models of emotion can be improved considerably by taking into account annotator agreement and training the model on smaller but reliable dataset. Finally we discuss the potential for improving prediction by combining the lexical and acoustic modalities. Simple fusion methods do not lead to consistent improvements over lexical classifiers alone but improve over acoustic models.

Vocalic, Lexical and Prosodic Cues for the INTERSPEECH 2018 Self-Assessed Affect Challenge

“…Lexical cues can be used to effectively discriminate the emotional valence from methods such as the Continuous Bag-Of-Words (CBOW) [31], the Pointwise Mutual Information (PMI) [21], the Term Frequency-Inverse Document Frequency (TFIDF), and the polarity lexicon [43]. Transcripts of audio clips are not available and were obtained through a voice dictation system.…”

“…For the second approach, the polarity score of each word of the Train set transcription has been computed using PMI method [21]. For this estimation, we use only the clips labeled low and high emotional valence.…”

“…A vast amount of literature is available about the acoustic and linguistic cues related to emotional speech [17,18,19,20,21,22,23]. The emotional speech databases were frequently based on play-acted speech.…”

“…However some studies [26,27,28,29,30] have reported that some parameters related to voice quality and speech articulation may be reliable with respect to identification of emotional valence: NAQ (Normalized Amplitude Quotient) [29], harmonic structure of the speech [30] such as HNR (Harmonic to Noise Ratio), Fraction of Locally Unvoiced Frames (FLUF) [27], glottis features [27], F2 and F3 formant values [26,29]. In a lexical approach [21,31], lexical cues related to emotional valence seemed more effective to discriminate emotional valence: Continuous Bag-Of-Words (CBOW) [31], Pointwise Mutual Information (PMI) [21], Term Frequency-Inverse Document Frequency (TFIDF), and polarity lexicon. For these experiments, transcription of speech was obtained by manual speech transcripts [21] or by Automatic Speech Recognition (ASR) system [19].…”

“…In a lexical approach [21,31], lexical cues related to emotional valence seemed more effective to discriminate emotional valence: Continuous Bag-Of-Words (CBOW) [31], Pointwise Mutual Information (PMI) [21], Term Frequency-Inverse Document Frequency (TFIDF), and polarity lexicon. For these experiments, transcription of speech was obtained by manual speech transcripts [21] or by Automatic Speech Recognition (ASR) system [19].…”