Weakly-Supervised Deep Convolutional Neural Network Learning for Facial Action Unit Intensity Estimation

Zhang, Yong; Dong, Weiming; Hu, Bao-Gang; Ji, Qiang

doi:10.1109/cvpr.2018.00246

Cited by 43 publications

(25 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Comparison with the State of the Art. We compared our model, referred to as SCC-Heatmap, to deep learningbased methods that leverage relationship modeling (CCNN-IT (Walecki et al, 2017), KBSS (Zhang et al, 2018a), BORMIR (Zhang et al, 2018b)) and the recent work KJRE (Zhang et al, 2019). Additionally, since our method is based on heatmap regression, we also directly applied two state-of-the-art regression models, Hourglass (Newell, Yang, and Deng, 2016) and ResNet-Deconv (Xiao, Wu, and Wei, 2018), for AU intensity estimation.…”

Section: Overall Performancementioning

confidence: 99%

Facial Action Unit Intensity Estimation via Semantic Correspondence Learning with Dynamic Graph Convolution

Fan

Lam

2020

AAAI

View full text Add to dashboard Cite

The intensity estimation of facial action units (AUs) is challenging due to subtle changes in the person's facial appearance. Previous approaches mainly rely on probabilistic models or predefined rules for modeling co-occurrence relationships among AUs, leading to limited generalization. In contrast, we present a new learning framework that automatically learns the latent relationships of AUs via establishing semantic correspondences between feature maps. In the heatmap regression-based network, feature maps preserve rich semantic information associated with AU intensities and locations. Moreover, the AU co-occurring pattern can be reflected by activating a set of feature channels, where each channel encodes a specific visual pattern of AU. This motivates us to model the correlation among feature channels, which implicitly represents the co-occurrence relationship of AU intensity levels. Specifically, we introduce a semantic correspondence convolution (SCC) module to dynamically compute the correspondences from deep and low resolution feature maps, and thus enhancing the discriminability of features. The experimental results demonstrate the effectiveness and the superior performance of our method on two benchmark datasets.

show abstract

Section: Overall Performancementioning

confidence: 99%

Facial Action Unit Intensity Estimation via Semantic Correspondence Learning with Dynamic Graph Convolution

Fan

Lam

2020

AAAI

View full text Add to dashboard Cite

show abstract

“…Let Given the partially annotated training set, our goal is to learn the parameters . In our previous work [58], as locations of keyframes of different AUs are different, we trained a model for each AU. In this extension, we introduce the task index as one input to train a model for joint intensity estimation of multiple AUs under the setting of semi-supervised learning.…”

Section: Proposed Approachmentioning

confidence: 99%

“…However, locations of keyframes of different AUs are different. As domain knowledge is applied to segments of individual AUs, in our previous work [58], we simply trained a network for each AU and made inference for each AU. To propose the efficiency of inference, we propose a strategy to train a network for multiple AUs by introducing the task index as one input of the network during training.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Semi-Supervised Deep Neural Network for Joint Intensity Estimation of Multiple Facial Action Units

et al. 2019

Self Cite

View full text Add to dashboard Cite

Facial action units (AUs) are defined to depict movements of facial muscles, which are basic elements to encode facial expressions. Automatic AU intensity estimation is an important task in affective computing. Previous works leverage the representation power of deep neural networks (DNNs) to improve the performance of intensity estimation. However, a large number of intensity annotations are required to train DNNs that contain millions of parameters. But it is expensive and difficult to build a largescale database with AU intensity annotation since AU annotation requires annotators have strong domain expertise. We propose a novel semi-supervised deep convolutional network that leverages extremely limited AU annotations for AU intensity estimation. It requires only intensity annotations of keyframes of training sequences. Domain knowledge on AUs is leveraged to provide weak supervisory information, including relative appearance similarity, temporal intensity ordering, facial symmetry, and contrastive appearance difference. We also propose a strategy to train a model for joint intensity estimation of multiple AUs under the setting of semi-supervised learning, which greatly improves the efficiency during inference. We perform empirical experiments on two public benchmark expression databases and make comparisons with state-ofthe-art methods to demonstrate the effectiveness of the proposed method.INDEX TERMS Facial action units, intensity estimation, deep learning, weakly supervised learning.

show abstract

“…Our goal in this paper is to devise a method that can effectively estimate AU intensity even when a very small (of the order of 2%) and randomly chosen set of frames is used for model training. This line of work has been pursued by the research community only recently [30,59,63,60,62,61]. While these works have shown remarkable results, they still have some limitations: 1) They work on a per-frame basis, by learning a strong image-based feature representation that can later be used for AU intensity estimation.…”

Section: Introductionmentioning

confidence: 99%

Semi-supervised Facial Action Unit Intensity Estimation with Contrastive Learning

Sánchez¹,

Bulat²,

Zaganidis³

et al. 2020

Preprint

View full text Add to dashboard Cite

This paper tackles the challenging problem of estimating the intensity of Facial Action Units with few labeled images. Contrary to previous works, our method does not require to manually select key frames, and produces state-of-the-art results with as little as 2% of annotated frames, which are randomly chosen. To this end, we propose a semisupervised learning approach where a spatio-temporal model combining a feature extractor and a temporal module are learned in two stages. The first stage uses datasets of unlabeled videos to learn a strong spatiotemporal representation of facial behavior dynamics based on contrastive learning. To our knowledge we are the first to build upon this framework for modeling facial behavior in an unsupervised manner. The second stage uses another dataset of randomly chosen labeled frames to train a regressor on top of our spatio-temporal model for estimating the AU intensity. We show that although backpropagation through time is applied only with respect to the output of the network for extremely sparse and randomly chosen labeled frames, our model can be effectively trained to estimate AU intensity accurately, thanks to the unsupervised pretraining of the first stage. We experimentally validate that our method outperforms existing methods when working with as little as 2% of randomly chosen data for both DISFA and BP4D datasets, without a careful choice of labeled frames, a time-consuming task still required in previous approaches.

show abstract

Weakly-Supervised Deep Convolutional Neural Network Learning for Facial Action Unit Intensity Estimation

Cited by 43 publications

References 38 publications

Facial Action Unit Intensity Estimation via Semantic Correspondence Learning with Dynamic Graph Convolution

Facial Action Unit Intensity Estimation via Semantic Correspondence Learning with Dynamic Graph Convolution

Semi-Supervised Deep Neural Network for Joint Intensity Estimation of Multiple Facial Action Units

Semi-supervised Facial Action Unit Intensity Estimation with Contrastive Learning

Contact Info

Product

Resources

About