An Investigation of Emotion Dynamics and Kalman Filtering for Speech-Based Emotion Prediction

Abstract

Despite recent interest in continuous prediction of dimensional emotions, the dynamical aspect of emotions has received less attention in automated systems. This paper investigates how emotion change can be effectively incorporated to improve continuous prediction of arousal and valence from speech. Significant correlations were found between emotion ratings and their dynamics during investigations on the RECOLA database, and here we examine how to best exploit them using a Kalman filter. In particular, we investigate the correlation between predicted arousal and valence dynamics with arousal and valence ground truth; the Kalman filter internal delay for estimating the state transition matrix; the use of emotion dynamics as a measurement input to a Kalman filter; and how multiple probabilistic Kalman filter outputs can be effectively fused. Evaluation results show that correct dynamics estimation and internal delay settings allow up to 5% and 58% relative improvement in arousal and valence prediction respectively over existing Kalman filter implementations. Fusion based on probabilistic Kalman filter outputs yields further gains.

Prediction of Emotion Change From Speech

“…The aim of this section is to explore system design and maximized system performance for a robust ECP, based on empirical experience in dealing with the emotion prediction problem. One may be interested in using emotion changes to facilitate absolute emotion predictions, such as Huang and Epps (2017) and Oveneke et al (2017) in which Kalman filtering was used. However, an important premise could be accurate predictions of emotion change, which is the only focus of this study.…”