Analysis of Emotionally Salient Aspects of Fundamental Frequency for Emotion Detection

Abstract

Abstract-During expressive speech, the voice is enriched to convey not only the intended semantic message but also the emotional state of the speaker. The pitch contour is one of the important properties of speech that is affected by this emotional modulation. Although pitch features have been commonly used to recognize emotions, it is not clear what aspects of the pitch contour are the most emotionally salient. This paper presents an analysis of the statistics derived from the pitch contour. First, pitch features derived from emotional speech samples are compared with the ones derived from neutral speech, by using symmetric Kullback-Leibler distance. Then, the emotionally discriminative power of the pitch features is quantified by comparing nested logistic regression models. The results indicate that gross pitch contour statistics such as mean, maximum, minimum, and range are more emotionally prominent than features describing the pitch shape. Also, analyzing the pitch statistics at the utterance level is found to be more accurate and robust than analyzing the pitch statistics for shorter speech regions (e.g., voiced segments). Finally, the best features are selected to build a binary emotion detection system for distinguishing between emotional versus neutral speech. A new two-step approach is proposed. In the first step, reference models for the pitch features are trained with neutral speech, and the input features are contrasted with the neutral model. In the second step, a fitness measure is used to assess whether the input speech is similar to, in the case of neutral speech, or different from, in the case of emotional speech, the reference models. The proposed approach is tested with four acted emotional databases spanning different emotional categories, recording settings, speakers and languages. The results show that the recognition accuracy of the system is over 77% just with the pitch features (baseline 50%). When compared to conventional classification schemes, the proposed approach performs better in terms of both accuracy and robustness.

Speaker-independent emotion recognition exploiting a psychologically-inspired binary cascade classification schema

“…Contrary to our work, the authors of [11] present a radically different approach, where just one feature category, namely pitch is sophisticatedly analysed. In a first step, reference pitch models for neutral speech are built.…”

“…The importance of pitch contour is exhaustively examined in [11]. Contrary to our work, the authors of [11] present a radically different approach, where just one feature category, namely pitch is sophisticatedly analysed.…”

“…On the other hand, several features are investigated here for the first time for the emotion recognition task. Mean of the 1st, 2nd, 3rd, and 4th formant [5][6][7][8] Maximum of the 1st, 2nd, 3rd, and 4th formant [9][10][11][12] Minimum of the 1st, 2nd, 3rd, and 4th formant [13][14][15][16] Variance of the 1st, 2nd, 3rd, and 4th formant [17][18][19][20] Skewness of the 1st, 2nd, 3rd, and 4th formant [21][22][23][24] Interquartile range of the 1st, 2nd, 3rd, and 4th formant [25][26][27][28] Range (i.e. maximum-minimum) of the 1st, 2nd, 3rd, and 4th formant [29][30][31][32] 90th percentile of the 1st, 2nd, 3rd, and 4th formant Several observations are made by the research community with respect to the correspondence among features and emotional categories.…”

“…The pitch contour is one of the important properties of speech that is affected by the emotional modulation [20] [25] [33] [43] [59], since emotional pitch modulation is triggered by the activation level of the sentence [11]. Previous work [11] has indicated that features such as mean, maximum, minimum, and range describe the global aspects of pitch contour and are more emotionally salient than features that describe the pitch shape itself, like slope, curvature, and inflexion.…”

“…Previous work [11] has indicated that features such as mean, maximum, minimum, and range describe the global aspects of pitch contour and are more emotionally salient than features that describe the pitch shape itself, like slope, curvature, and inflexion. Energy, formants, voice quality, and spectral features efficiency is already verified [15] [49].…”