Spatio-Temporal Encoder-Decoder Fully Convolutional Network for Video-Based Dimensional Emotion Recognition

Du, Zhengyin; Wu, Suowei; Huang, Di; Li, Weixin; Wang, Yunlong

doi:10.1109/taffc.2019.2940224

Cited by 28 publications

(10 citation statements)

References 62 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The experimental results demonstrated that the LSTM outperformed SVR dramatically. Compared with RNN based methods, a fully convolutional network was designed by Du et al [6]. It performed emotion recognition in a coarse-to-fine strategy by aggregating multi-level features with various scales.…”

Section: Related Workmentioning

confidence: 99%

Hybrid Mutimodal Fusion for Dimensional Emotion Recognition

Sun

et al. 2021

Proceedings of the 2nd on Multimodal Sentiment Analysis Challenge

View full text Add to dashboard Cite

In this paper, we extensively present our solutions for the MuSe-Stress sub-challenge and the MuSe-Physio sub-challenge of Multimodal Sentiment Challenge (MuSe) 2021. The goal of MuSe-Stress sub-challenge is to predict the level of emotional arousal and valence in a time-continuous manner from audio-visual recordings and the goal of MuSe-Physio sub-challenge is to predict the level of psycho-physiological arousal from a) human annotations fused with b) galvanic skin response (also known as Electrodermal Activity (EDA)) signals from the stressed people. The Ulm-TSST dataset which is a novel subset of the audio-visual textual Ulm-Trier Social Stress dataset that features German speakers in a Trier Social Stress Test (TSST) induced stress situation is used in both subchallenges. For the MuSe-Stress sub-challenge, we highlight our solutions in three aspects: 1) the audio-visual features and the biosignal features are used for emotional state recognition. 2) the Long Short-Term Memory (LSTM) with the self-attention mechanism is utilized to capture complex temporal dependencies within the feature sequences. 3) the late fusion strategy is adopted to further boost the model's recognition performance by exploiting complementary information scattered across multimodal sequences. Our proposed model achieves CCC of 0.6159 and 0.4609 for valence and arousal respectively on the test set, which both rank in the top 3. For the MuSe-Physio sub-challenge, we first extract the audiovisual features and the bio-signal features from multiple modalities. Then, the LSTM module with the self-attention mechanism, and the Gated Convolutional Neural Networks (GCNN) as well as the * Both authors contributed equally to this research.

show abstract

Section: Related Workmentioning

confidence: 99%

Hybrid Mutimodal Fusion for Dimensional Emotion Recognition

Sun

et al. 2021

Proceedings of the 2nd on Multimodal Sentiment Analysis Challenge

View full text Add to dashboard Cite

show abstract

“…Temporal Modelling in Emotion Recognition: Most existing methods model the temporal dynamics of continuous emotions using deterministic approaches such as Time-Delay Neural Networks [43], RNNs, LSTMs and GRUs [32,28,26,71,8,59], multi-head attention models [20], 3D Convolutions [76,35], 3D ConvLSTMs [19], and temporal-hourglass CNNs [10]. While these deterministic models are capable of effectively learning the temporal dynamics, they do not take the inherent stochastic nature of the continuous emotion labels into account.…”

Section: Related Workmentioning

confidence: 99%

Affective Processes: stochastic modelling of temporal context for emotion and facial expression recognition

Sánchez¹,

Tellamekala²,

Valstar³

et al. 2021

Preprint

View full text Add to dashboard Cite

Temporal context is key to the recognition of expressions of emotion. Existing methods, that rely on recurrent or selfattention models to enforce temporal consistency, work on the feature level, ignoring the task-specific temporal dependencies, and fail to model context uncertainty. To alleviate these issues, we build upon the framework of Neural Processes to propose a method for apparent emotion recognition with three key novel components: (a) probabilistic contextual representation with a global latent variable model; (b) temporal context modelling using task-specific predictions in addition to features; and (c) smart temporal context selection. We validate our approach on four databases, two for Valence and Arousal estimation (SEWA and AffWild2), and two for Action Unit intensity estimation (DISFA and BP4D). Results show a consistent improvement over a series of strong baselines as well as over state-ofthe-art methods.

show abstract

“…Moreover, it is difficult for LSTM-RNNs to avoid the exploding/vanishing gradient problem when input sequences are long [9]. Recently, due to their advantageous parallelism, flexible receptive field and stable gradient, temporal convolutional networks [8] have proven effective at capturing long range patterns [10].…”

Section: Introductionmentioning

confidence: 99%

Hierarchical Attention-Based Temporal Convolutional Networks for Eeg-Based Emotion Recognition

Chen

Zhao

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

EEG-based emotion recognition is an effective way to infer the inner emotional state of human beings. Recently, deep learning methods, particularly long short-term memory recurrent neural networks (LSTM-RNNs), have made encouraging progress for in the field of emotion recognition. However, the LSTM-RNNs are time-consuming and have difficulty avoiding the problem of exploding/vanishing gradients when during training. In addition, EEG-based emotion recognition often suffers due to the existence of silent and emotional irrelevant frames from intra-channel. Not all channels carry the same emotional discriminative information. In order to tackle these problems, a hierarchical attention-based temporal convolutional networks (HATCN) for efficient EEG-based emotion recognition is proposed. Firstly, a spectrogram representation is generated from raw EEG signals in each channel to capture their time and frequency information. Secondly, temporal convolutional networks (TCNs) are utilised to automatically learn more robust/intrinsic long-term dynamic characters in emotion response. Next, a hierarchical attention mechanism is investigated that aggregates the emotional information at both the frame and channel level. The experimental results on the DEAP dataset show that our method achieves an average recognition accuracy of 0.716 and an F1-score of 0.642 over four emotional dimensions and outperforms other state-of-the-art methods in a user-independent scenario.

show abstract

Spatio-Temporal Encoder-Decoder Fully Convolutional Network for Video-Based Dimensional Emotion Recognition

Cited by 28 publications

References 62 publications

Hybrid Mutimodal Fusion for Dimensional Emotion Recognition

Hybrid Mutimodal Fusion for Dimensional Emotion Recognition

Affective Processes: stochastic modelling of temporal context for emotion and facial expression recognition

Hierarchical Attention-Based Temporal Convolutional Networks for Eeg-Based Emotion Recognition

Contact Info

Product

Resources

About