A Transfer Learning approach to Heatmap Regression for Action Unit intensity estimation

Ntinou, Ioanna; Sánchez, Enrique; Bulat, Adrian; Valstar, Michel; Tzimiropoulos, Georgios

doi:10.48550/arxiv.2004.06657

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most recent methods reporting state of the art results on Action Unit intensity estimation build on AutoEncoders [44], or on Heatmap Regression [41,35]. While there is a vast amount of literature in fully supervised methods for Action Unit intensity estimation, few works have focused on the more challenging task of addressing the same goal in a semi-supervised manner.…”

Section: Action Unit Modelingmentioning

confidence: 99%

Semi-supervised Facial Action Unit Intensity Estimation with Contrastive Learning

Sánchez¹,

Bulat²,

Zaganidis³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

This paper tackles the challenging problem of estimating the intensity of Facial Action Units with few labeled images. Contrary to previous works, our method does not require to manually select key frames, and produces state-of-the-art results with as little as 2% of annotated frames, which are randomly chosen. To this end, we propose a semisupervised learning approach where a spatio-temporal model combining a feature extractor and a temporal module are learned in two stages. The first stage uses datasets of unlabeled videos to learn a strong spatiotemporal representation of facial behavior dynamics based on contrastive learning. To our knowledge we are the first to build upon this framework for modeling facial behavior in an unsupervised manner. The second stage uses another dataset of randomly chosen labeled frames to train a regressor on top of our spatio-temporal model for estimating the AU intensity. We show that although backpropagation through time is applied only with respect to the output of the network for extremely sparse and randomly chosen labeled frames, our model can be effectively trained to estimate AU intensity accurately, thanks to the unsupervised pretraining of the first stage. We experimentally validate that our method outperforms existing methods when working with as little as 2% of randomly chosen data for both DISFA and BP4D datasets, without a careful choice of labeled frames, a time-consuming task still required in previous approaches.

show abstract

Section: Action Unit Modelingmentioning

confidence: 99%