Multimodal Emotion Recognition for AVEC 2016 Challenge

Povolny, Filip; Matějka, Pavel; Hradiš, Michal; Popková, Anna; Otrusina, Lubomir; Smrž, Pavel; Wood, I. G.; Robin, Cécile; Lamel, Lori

doi:10.1145/2988257.2988268

Cited by 46 publications

(40 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Povolny et al used all features to train linear regressors to predict a value for each frame, and considered two methods for incorporating contextual information: simple frame stacking and temporal content summarization by applying statistics to local windows. In contrast, in this work we show that considering temporal dependencies that are longer than those presented in [6,7] is critical to improve continuous emotion recognition performance.…”

Section: Related Workmentioning

confidence: 58%

“…The higher-level audio features were used to train linear SVRs. Povolny et al [7] used eGeMAPS [8] features along with a set of higher-level bottleneck features extracted from a DNN trained for automatic speech recognition (ASR) to train linear regressors. The higher level features were produced from an initial set of 24 Mel filterbank (MFB) features and four different estimates of the fundamental frequency (F0).…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Capturing Long-Term Temporal Dependencies with Convolutional Networks for Continuous Emotion Recognition

Khorram

Aldeneh

Dimitriadis³

et al. 2017

Interspeech 2017

View full text Add to dashboard Cite

The goal of continuous emotion recognition is to assign an emotion value to every frame in a sequence of acoustic features. We show that incorporating long-term temporal dependencies is critical for continuous emotion recognition tasks. To this end, we first investigate architectures that use dilated convolutions. We show that even though such architectures outperform previously reported systems, the output signals produced from such architectures undergo erratic changes between consecutive time steps. This is inconsistent with the slow moving ground-truth emotion labels that are obtained from human annotators. To deal with this problem, we model a downsampled version of the input signal and then generate the output signal through upsampling. Not only does the resulting downsampling/upsampling network achieve good performance, it also generates smooth output trajectories. Our method yields the best known audioonly performance on the RECOLA dataset.

show abstract

Section: Related Workmentioning

confidence: 58%

Section: Related Workmentioning

confidence: 99%

Capturing Long-Term Temporal Dependencies with Convolutional Networks for Continuous Emotion Recognition

Khorram

Aldeneh

Dimitriadis³

et al. 2017

Interspeech 2017

View full text Add to dashboard Cite

show abstract

“…The most directly comparable system is a deep BLSTM-RNN regressor trained with MSE loss, as both MSE and CCE are frame-level objectives. Previous works have reported that using CCC as the objective function consistently outperforms MSE for regression models [16,17]. In addition to being more in line with the evaluation metric, CCC has the advantage over MSE and CCE in that it is an utterance-level objective that takes into account the overall shape of the time series.…”

Section: Deep Blstm-rnn Regressormentioning

confidence: 99%

“…The best performer of AVEC 2016, Brady et al, used SVR trained on sparse-coded higherlevel representations of various types of audio features [15]. Povolny et al trained a set of linear regressors on eGeMAPS augmented with deep bottleneck features from Deep Neural Network (DNN) acoustic models [16]. Trigeorgis et al trained a convolutional RNN directly on raw waveform [17].…”

Section: Avec Approachesmentioning

confidence: 99%

Discretized Continuous Speech Emotion Recognition with Multi-Task Deep Recurrent Neural Network

Aldeneh

Provost

2017

Interspeech 2017

View full text Add to dashboard Cite

Estimating continuous emotional states from speech as a function of time has traditionally been framed as a regression problem. In this paper, we present a novel approach that moves the problem into the classification domain by discretizing the training labels at different resolutions. We employ a multi-task deep bidirectional long-short term memory (BLSTM) recurrent neural network (RNN) trained with cost-sensitive Cross Entropy loss to model these labels jointly. We introduce an emotion decoding algorithm that incorporates long-and short-term temporal properties of the signal to produce more robust time series estimates. We show that our proposed approach achieves competitive audio-only performance on the RECOLA dataset, relative to previously published works as well as other strong regression baselines. This work provides a link between regression and classification, and contributes an alternative approach for continuous emotion recognition.

show abstract

“…Some people cry out feeling sad, while others keep a neutral expression in an attempt to hide their true feelings [23], adding to the difficulty of recognising emotions. Research on previous AVEC challenges have explored various methods: extracting multimodal features [6,35], the fusion of multiple modalities [8,34] and numerous deep learning architectures [33,43] for recognising emotions.…”

Section: Introductionmentioning

confidence: 99%

Towards a Better Gold Standard

Wang

Lopes

Pun

et al. 2018

Proceedings of the 2018 on Audio/Visual Emotion Challenge and Workshop

View full text Add to dashboard Cite

Emotions are often perceived by humans through a series of multimodal cues, such as verbal expressions, facial expressions and gestures. In order to recognise emotions automatically, reliable emotional labels are required to learn a mapping from human expressions to corresponding emotions. Dimensional emotion models have become popular and have been widely applied for annotating emotions continuously in the time domain. However, the statistical relationship between emotional dimensions is rarely studied. This paper provides a solution to automatic emotion recognition for the Audio/Visual Emotion Challenge (AVEC) 2018. The objective is to find a robust way to detect emotions using more reliable emotion annotations in the valence and arousal dimensions. The two main contributions of this paper are: 1) the proposal of a new approach capable of generating more dependable emotional ratings for both arousal and valence from multiple annotators by extracting consistent annotation features; 2) the exploration of the valence and arousal distribution using outlier detection methods, which shows a specific oblique elliptic shape. With the learned distribution, we are able to detect the prediction outliers based on their local density deviations and correct them towards the learned distribution. The proposed method performance is evaluated on the RECOLA database containing audio, video and physiological recordings. Our results show that a moving average filter is sufficient to remove the incidental errors in annotations. The unsupervised dimensionality reduction approaches could be used to determine a gold standard annotations from multiple annotations. Compared with the baseline model of AVEC 2018, our approach improved the arousal and valence prediction of concordance correlation coefficient significantly to respectively 0.821 and 0.589.

show abstract

Multimodal Emotion Recognition for AVEC 2016 Challenge

Cited by 46 publications

References 16 publications

Capturing Long-Term Temporal Dependencies with Convolutional Networks for Continuous Emotion Recognition

Capturing Long-Term Temporal Dependencies with Convolutional Networks for Continuous Emotion Recognition

Discretized Continuous Speech Emotion Recognition with Multi-Task Deep Recurrent Neural Network

Towards a Better Gold Standard

Contact Info

Product

Resources

About