Temporally Coherent Visual Representations for Dimensional Affect Recognition

Tellamekala, Mani Kumar; Valstar, Michel

doi:10.1109/acii.2019.8925529

Cited by 14 publications

(9 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the current AU labelled video datasets are limited in terms of the total duration and the number of subjects. To address this significant challenge, leveraging the naturally available supervision cues, such as temporal coherency of facial actions in a video [62], [63], is an alternative approach worth considering to make the 3D face more expressive in a label-efficient manner.…”

Section: Summary and Discussionmentioning

confidence: 99%

Are 3D Face Shapes Expressive Enough for Recognising Continuous Emotions and Action Unit Intensities?

Tellamekala¹,

Sümer²,

Schuller³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Recognising continuous emotions and action unit (AU) intensities from face videos requires a spatial and temporal understanding of expression dynamics. Existing works primarily rely on 2D face appearances to extract such dynamics. This work focuses on a promising alternative based on parametric 3D face shape alignment models, which disentangle different factors of variation, including expression-induced shape variations. We aim to understand how expressive 3D face shapes are in estimating valence-arousal and AU intensities compared to the state-of-the-art 2D appearance-based models. We benchmark four recent 3D face alignment models: ExpNet, 3DDFA-V2, DECA, and EMOCA. In valence-arousal estimation, expression features of 3D face models consistently surpassed previous works and yielded an average concordance correlation of .739 and .574 on SEWA and AVEC 2019 CES corpora, respectively. We also study how 3D face shapes performed on AU intensity estimation on BP4D and DISFA datasets, and report that 3D face features were on par with 2D appearance features in AUs 4, 6, 10, 12, and 25, but not the entire set of AUs. To understand this discrepancy, we conduct a correspondence analysis between valence-arousal and AUs, which points out that accurate prediction of valence-arousal may require the knowledge of only a few AUs.

show abstract

Section: Summary and Discussionmentioning

confidence: 99%

Are 3D Face Shapes Expressive Enough for Recognising Continuous Emotions and Action Unit Intensities?

Tellamekala¹,

Sümer²,

Schuller³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Temporal Modelling in Emotion Recognition: Most existing methods model the temporal dynamics of continuous emotions using deterministic approaches such as Time-Delay Neural Networks [43], RNNs, LSTMs and GRUs [32,28,26,71,8,59], multi-head attention models [20], 3D Convolutions [76,35], 3D ConvLSTMs [19], and temporal-hourglass CNNs [10]. While these deterministic models are capable of effectively learning the temporal dynamics, they do not take the inherent stochastic nature of the continuous emotion labels into account.…”

Section: Related Workmentioning

confidence: 99%

Affective Processes: stochastic modelling of temporal context for emotion and facial expression recognition

Sánchez¹,

Tellamekala²,

Valstar³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Temporal context is key to the recognition of expressions of emotion. Existing methods, that rely on recurrent or selfattention models to enforce temporal consistency, work on the feature level, ignoring the task-specific temporal dependencies, and fail to model context uncertainty. To alleviate these issues, we build upon the framework of Neural Processes to propose a method for apparent emotion recognition with three key novel components: (a) probabilistic contextual representation with a global latent variable model; (b) temporal context modelling using task-specific predictions in addition to features; and (c) smart temporal context selection. We validate our approach on four databases, two for Valence and Arousal estimation (SEWA and AffWild2), and two for Action Unit intensity estimation (DISFA and BP4D). Results show a consistent improvement over a series of strong baselines as well as over state-ofthe-art methods.

show abstract

“…The recent growth of video-based datasets has encouraged the inclusion of temporal modelling, which has shown to improve models' training (Xie et al, 2016;Cootes et al, 1998). Relevant examples in Affective Computing include the works of Tellamekala et al (Tellamekala and Valstar, 2019) and Ma et al (Ma et al, 2019). In their work, Tellamekala et al (Tellamekala and Valstar, 2019) enforce temporal coherency and smoothness aspects on their feature representation by constraining the differences between adjacent frames, while Ma et al resort to the utilisation of LSTM RNNs with residual connections applied to multi-modal data.…”

Section: Related Workmentioning

confidence: 99%

“…Relevant examples in Affective Computing include the works of Tellamekala et al (Tellamekala and Valstar, 2019) and Ma et al (Ma et al, 2019). In their work, Tellamekala et al (Tellamekala and Valstar, 2019) enforce temporal coherency and smoothness aspects on their feature representation by constraining the differences between adjacent frames, while Ma et al resort to the utilisation of LSTM RNNs with residual connections applied to multi-modal data. Furthermore, the use of attention has also been recently explored by Xiaohua et al (Xiaohua et al, 2019) and Li et al (Li et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

“…Further, the emergence of video-based data allows to enrich the widely used spatial features with the inclusion of temporal information. To this end, several authors have explored the use of long-short term a https://orcid.org/0000-0002-6653-3470 b https://orcid.org/0000-0002-2029-1576 c https://orcid.org/0000-0002-6478-8699 d https://orcid.org/0000-0002-4324-9952 memory (LSTM) recurrent neural networks (RNNs) (Tellamekala and Valstar, 2019;Ma et al, 2019), endowed also with attention mechanisms (Luong et al, 2015;Li et al, 2020;Xiaohua et al, 2019). However, such spatio-temporal modelling often results in very high-dimensional feature spaces and large volumes of data, making training difficult and time consuming.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

An Enhanced Adversarial Network with Combined Latent Features for Spatio-Temporal Facial Affect Estimation in the Wild

Aspandi,

Sukno,

Schuller

et al. 2021

Preprint

View full text Add to dashboard Cite

Affective Computing has recently attracted the attention of the research community, due to its numerous applications in diverse areas. In this context, the emergence of video-based data allows to enrich the widely used spatial features with the inclusion of temporal information. However, such spatio-temporal modelling often results in very high-dimensional feature spaces and large volumes of data, making training difficult and time consuming. This paper addresses these shortcomings by proposing a novel model that efficiently extracts both spatial and temporal features of the data by means of its enhanced temporal modelling based on latent features. Our proposed model consists of three major networks, coined Generator, Discriminator, and Combiner, which are trained in an adversarial setting combined with curriculum learning to enable our adaptive attention modules. In our experiments, we show the effectiveness of our approach by reporting our competitive results on both the AFEW-VA and SEWA datasets, suggesting that temporal modelling improves the affect estimates both in qualitative and quantitative terms. Furthermore, we find that the inclusion of attention mechanisms leads to the highest accuracy improvements, as its weights seem to correlate well with the appearance of facial movements, both in terms of temporal localisation and intensity. Finally, we observe the sequence length of around 160 ms to be the optimum one for temporal modelling, which is consistent with other relevant findings utilising similar lengths.

show abstract

Temporally Coherent Visual Representations for Dimensional Affect Recognition

Cited by 14 publications

References 23 publications

Are 3D Face Shapes Expressive Enough for Recognising Continuous Emotions and Action Unit Intensities?

Are 3D Face Shapes Expressive Enough for Recognising Continuous Emotions and Action Unit Intensities?

Affective Processes: stochastic modelling of temporal context for emotion and facial expression recognition

An Enhanced Adversarial Network with Combined Latent Features for Spatio-Temporal Facial Affect Estimation in the Wild

Contact Info

Product

Resources

About