An Investigation of Annotation Delay Compensation and Output-Associative Fusion for Multimodal Continuous Emotion Prediction

Huang, Zhaocheng; Dang, Ting; Cummins, Nicholas; Stasak, Brian; Le, Phu Ngoc; Sethu, Vidhyasaharan; Epps, Julien

doi:10.1145/2808196.2811640

Cited by 66 publications

(70 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other hand, delays for absolute emotions vary for arousal and valence as well as for different databases. The delays for absolute affect dimensions yielded similar results as previous studies on RECOLA (Huang et al, 2015a) and SEMAINE (Nicolle et al, 2012;Mariooryad and Busso, 2014).…”

Section: Delay Compensationsupporting

confidence: 84%

“…The OA-RVM performances in ρ c were higher than the audio-only results but somewhat lower than the multimodal results in Huang et al (2015a) on the RECOLA dataset. For SEMAINE, the OA-RVM performances were much lower in Pearson's correlation ρ when compared with the winners of the AVEC 2012 challenge (Nicolle et al, 2012), who achieve 0.65 (arousal) and 0.33 (valence) on development set, 0.61(arousal), and 0.34 (valence) on test set.…”

Section: Delay Compensationcontrasting

confidence: 55%

“…This enables the framework to take advantage of the following dependencies: (1) between arousal and valence predictions; (2) between previous and future predictions; and (3) between output predictions and input features. In addition, it is robust for different system configurations while providing state-of-the-art performances (Huang et al, 2015a). Detailed descriptions of SVR, RVM and OA RVM can be referred to in , Tipping (2001) and Nicolaou et al (2012).…”

Section: Experimental System Settingsmentioning

confidence: 99%

“…Dynamic Time Warping (DTW) was also been found effective to eliminate inter-rater delays and outliers (Katsimerou et al, 2015). Another method proposed in Huang et al (2015a) for compensating delays is to apply temporal shifts of features and ratings to build reliable models in the training phase. Delays introduced in predictions were further compensated via a binomial filter, which, in addition, smooths predicted affect dimensions.…”

Section: Delay Compensationmentioning

confidence: 99%

See 3 more Smart Citations

Prediction of Emotion Change From Speech

Huang

Epps

2018

Front. ICT

Self Cite

View full text Add to dashboard Cite

The fact that emotions are dynamic in nature and evolve across time has been explored relatively less often in automatic emotion recognition systems to date. Although withinutterance information about emotion changes recently has received some attention, there remain open questions unresolved, such as how to approach delta emotion ground truth, how to predict the extent of emotion change from speech, and how well change can be predicted relative to absolute emotion ratings. In this article, we investigate speech-based automatic systems for continuous prediction of the extent of emotion changes in arousal/valence. We propose the use of regression (smoothed) deltas as ground truth for emotion change, which yielded considerably higher inter-rater reliability than first-order deltas, a commonly used approach in previous research, and represent a more appropriate approach to derive annotations for emotion change research, findings which are applicable beyond speech-based systems. In addition, the first system design for continuous emotion change prediction from speech is explored. Experimental results under the Output-Associative Relevance Vector Machine framework interestingly show that changes in emotion ratings may be better predicted than absolute emotion ratings on the RECOLA database, achieving 0.74 vs. 0.71 for arousal and 0.41 vs. 0.37 for valence in concordance correlation coefficients. However, further work is needed to achieve effective emotion change prediction performances on the SEMAINE database, due to the large number of non-change frames in the absolute emotion ratings.

show abstract

Section: Delay Compensationsupporting

confidence: 84%

Section: Delay Compensationcontrasting

confidence: 55%

Section: Experimental System Settingsmentioning

confidence: 99%

Section: Delay Compensationmentioning

confidence: 99%

See 2 more Smart Citations

Prediction of Emotion Change From Speech

Huang

Epps

2018

Front. ICT

Self Cite

View full text Add to dashboard Cite

show abstract

“…One common approach when considering multiple modalities is early (aka feature-level) fusion of unimodal information. This is typically achieved by concatenating all the features from multiple modalities into one combined feature vector, which is then used as the input information for the models [16], [43]- [45]. A benefit of early fusion is that it can provide better discriminative ability to the model by exploiting the complementary information that exists among different modalities.…”

Section: Related Workmentioning

confidence: 99%

Dynamic Difficulty Awareness Training for Continuous Emotion Prediction

Zhang

Han

Coutinho

et al. 2019

IEEE Trans. Multimedia

View full text Add to dashboard Cite

Time-continuous emotion prediction has become an increasingly compelling task in machine learning. Considerable efforts have been made to advance the performance of these systems. Nonetheless, the main focus has been the development of more sophisticated models and the incorporation of different expressive modalities (e. g., speech, face, and physiology). In this paper, motivated by the benefit of difficulty awareness in a human learning procedure, we propose a novel machine learning framework, namely, Dynamic Difficulty Awareness Training (DDAT), which sheds fresh light on the research -directly exploiting the difficulties in learning to boost the machine learning process. The DDAT framework consists of two stages: information retrieval and information exploitation. In the first stage, we make use of the reconstruction error of input features or the annotation uncertainty to estimate the difficulty of learning specific information. The obtained difficulty level is then used in tandem with original features to update the model input in a second learning stage with the expectation that the model can learn to focus on high difficulty regions of the learning process. We perform extensive experiments on a benchmark database (RECOLA) to evaluate the effectiveness of the proposed framework. The experimental results show that our approach outperforms related baselines as well as other well-established time-continuous emotion prediction systems, which suggests that dynamically integrating the difficulty information for neural networks can help enhance the learning process.

show abstract