Spatio-Temporal Relation and Attention Learning for Facial Action Unit Detection

Abstract

Spatio-temporal relations among facial action units (AUs) convey significant information for AU detection yet have not been thoroughly exploited. The main reasons are the limited capability of current AU detection works in simultaneously learning spatial and temporal relations, and the lack of precise localization information for AU feature learning. To tackle these limitations, we propose a novel spatio-temporal relation and attention learning framework for AU detection. Specifically, we introduce a spatiotemporal graph convolutional network to capture both spatial and temporal relations from dynamic AUs, in which the AU relations are formulated as a spatio-temporal graph with adaptively learned instead of predefined edge weights. Moreover, the learning of spatio-temporal relations among AUs requires individual AU features. Considering the dynamism and shape irregularity of AUs, we propose an attention regularization method to adaptively learn regional attentions that capture highly relevant regions and suppress irrelevant regions so as to extract a complete feature for each AU. Extensive experiments show that our approach achieves substantial improvements over the stateof-the-art AU detection methods on BP4D and especially DISFA benchmarks.

Spatio-Temporal Analysis of Facial Actions using Lifecycle-Aware Capsule Networks

“…Popular methods include using geometric features that track facial landmarks [7], histogram-based approaches that cluster local features into uniform regions for processing [32] or using features that describe local neighbourhoods [2]. With the popularity of deep learning, CNN [11,14,21] and graph-based [19,36] approaches have achieved stateof-the-art results for AU detection due to their ability to hierarchically learn spatial features. Capsule-based computations [31] offer an improvement as along with learning to detect different facial features, they also learn how these are arranged with respect to each other.…”

“…However, their approach focuses on extracting spatio-temporal features from complete video sequences at once, dropping certain adjacent frames to ensure all video sequences are of the same length. Other more recent approaches learn semantic relationships between different face regions and represent these using structured knowledge-graphs, learning coupling patterns between regions using graph-based computations [19,36].…”

“…Computational models for AU detection, traditionally, have explored local spatial relationships and hierarchies between different face regions using shape-based representations or using spectral or histogram-based methods [32,44]. With deep learning gaining popularity, recent approaches [4,14,21,35,36,46,47] have applied convolution or graph-based models to focus on learning such facial features directly from data, outperforming most traditional approaches. More recently, capsule-based computations proposed by Sabour et al [31] have further improved the learning of spatial dependencies in the form of facial feature primitives.…”

“…As a result, even though these approaches are able to detect strong AU activations in posed settings or when an expression is highly accentuated, they suffer when detecting more subtle expressions in spontaneous and naturalistic settings [38,42], challenging their real-world applicability. A prevailing requirement for automatic AU detection is to be sensitive to the said AU lifecycle and include temporal information, such as motion features or correlations amongst proximal frames, along with spatial features [19,36,42]. While spatial processing is important to determine relationships between different facial regions [19], understanding temporal correlations between their activation patterns in contiguous frames provides essential information about the AU lifecycle and can be particularly useful in detecting subtle activations [4,36,42].…”

“…A prevailing requirement for automatic AU detection is to be sensitive to the said AU lifecycle and include temporal information, such as motion features or correlations amongst proximal frames, along with spatial features [19,36,42]. While spatial processing is important to determine relationships between different facial regions [19], understanding temporal correlations between their activation patterns in contiguous frames provides essential information about the AU lifecycle and can be particularly useful in detecting subtle activations [4,36,42].…”