Correcting Time-Continuous Emotional Labels by Modeling the Reaction Lag of Evaluators

Abstract

An appealing scheme to characterize expressive behaviors is the use of emotional dimensions such as activation (calm versus active) and valence (negative versus positive). These descriptors offer many advantages to describe the wide spectrum of emotions. Due to the continuous nature of fast-changing expressive vocal and gestural behaviors, it is desirable to continuously track these emotional traces, capturing subtle and localized events (e.g., with FEELTRACE). However, time-continuous annotations introduce challenges that affect the reliability of the labels. In particular, an important issue is the evaluators' reaction lag caused by observing, appraising, and responding to the expressive behaviors. An empirical analysis demonstrates that this delay varies from 1 to 6 seconds, depending on the annotator, expressive dimension, and actual behaviors. Our experiments show accuracy improvements even with fixed delays (1-3 seconds). This paper proposes to compensate for this reaction lag by finding the time-shift that maximizes the mutual information between the expressive behaviors and the time-continuous annotations. The approach is implemented by making different assumptions about the evaluators' reaction lag. The benefits of compensating for the delay is demonstrated with emotion classification experiments. On average, the classifiers trained with facial and speech features show more than 7 percent relative improvements over baseline classifiers trained and tested without shifting the time-continuous annotations.

Continuous Estimation of Emotions in Speech by Dynamic Cooperative Speaker Models

“…Results show that an average RL of 3.89 s is obtained for arousal (σ = 1.16 s) and 4.52 s (σ = 2.15 s) for valence, in total agreement with the experimental results reported in the literature [24], [26]. Arousal is indeed a less subjective emotion than valence and thus requires less time for being evaluated.…”

“…Additionally, humans have natural bias and inconsistencies in their judgement [23], which creates additional noise in the ratings. Further, the variability in emotion perception can also be observed in the time domain, since the evaluators may have different reaction lag (RL) during the procedure of time-continuous annotation [24]. However, the natural diversity found in emotion perception is usually merged when a machine learning model is trained, by averaging several evaluations from a pool of raters into a single gold standard.…”

“…The issue of synchronisation of various individual ratings for defining a gold standard has also been investigated with signal processing techniques. Models of RL have been estimated from the data, by maximising the correlation coefficient ( [27], [28], [29]), or the mutual information ( [24]) between audiovisual features and emotional ratings while shifting back in time the latter.…”

“…This is especially observable on time-continuous ratings used on dimensional models of emotion, where a delay occurs between the observable cues and the reported emotional value. According to the evaluations performed in [24], we assume here a RL distinct for each speaker and emotional dimension with a negligible variation among the six ratings of the same speaker, compensating this effect with the correlation-based estimation of the gold standard. However, we relate the estimation of the optimal RL to a feature selection procedure that is performed independently on each quadrant of the 2D arousal-valence emotional space, to consider the peculiarities of the acoustic features according to the emotions.…”

Can We Generate Emotional Pronunciations for Expressive Speech Synthesis?

“…According to Campbell [3], "part of the reason for the dominance of discrete emotions is the ease of collecting training data". In order to take into consideration other aspects of expressive voice such as social cues, intention or interactive cues, the complex nature of affect in speech can be described with continuous dimensions, notably activation and valence [7], but also control, dominance or intention [8].…”