Cross-domain AU Detection: Domains, Learning Approaches, and Measures

Abstract

Facial action unit (AU) detectors have performed well when trained and tested within the same domain. Do AU detectors transfer to new domains in which they have not been trained? To answer this question, we review literature on cross-domain transfer and conduct experiments to address limitations of prior research. We evaluate both deep and shallow approaches to AU detection (CNN and SVM, respectively) in two large, well-annotated, publicly available databases, Expanded BP4D+ and GFT. The databases differ in observational scenarios, participant characteristics, range of head pose, video resolution, and AU base rates. For both approaches and databases, performance decreased with change in domain, often to below the threshold needed for behavioral research. Decreases were not uniform, however. They were more pronounced for GFT than for Expanded BP4D+ and for shallow relative to deep learning. These findings suggest that more varied domains and deep learning approaches may be better suited for promoting generalizability. Until further improvement is realized, caution is warranted when applying AU classifiers from one domain to another.

D-PAttNet: Dynamic Patch-Attentive Deep Network for Action Unit Detection

“…More generally, generalizability of models and decision thresholds across databases or domains are open research questions. Decreases in classifier performance are common in cross-domain settings (Onal Ertugrul et al, 2019a) even when models are trained on large databases. Future work should explore cross-domain generalizability of models and thresholds in large databases that vary in pose characteristics.…”

“…Traditional AU detection methods are based on (i) extracting appearance (Jiang et al, 2011;Eleftheriadis et al, 2015;Baltrusaitis et al, 2018) or geometric features (Lucey et al, 2007;Du et al, 2014) from the whole face and (ii) obtaining shallow representations as histograms of these features, thus ignoring the specificity of facial parts to AUs (Shojaeilangari et al, 2015). Deep approaches using whole face to train CNNs (Hammal et al, 2017;Onal Ertugrul et al, 2019a) also ignore the specificity of facial parts. More recent approaches focus on obtaining local representations using patch learning.…”

Assessing Automated Facial Action Unit Detection Systems for Analyzing Cross-Domain Facial Expression Databases

“…Because manual FACS coding has the disadvantage of being time consuming, automatic detection of FACS AUs has been an active area of research [ 5 ]. Automated facial AU detection systems are available as both commercial tools (e.g., Affectiva, FaceReader) and open-source tools (e.g., OpenFace, [ 6 , 7 ] and Automated Facial Affect Recognition (AFAR) [ 8 , 9 ]). One study found that OpenFace and AFAR generally performed similarly, but average results were slightly better for AFAR [ 5 ].…”

“…Second, there has been no systematic comparison of AU detection accuracy among systems. FaceReader is a commercial software designed to analyze facial expressions, whereas OpenFace [ 6 , 7 ] is the dominant shareware automatic facial computing system for many applied situations [ 17 , 18 ], and AFAR is an open-source, state-of-the-art, algorithm-based user-friendly tool for automated AU detection [ 8 , 9 ]. Although comparisons of the performance of these systems are interesting for newcomers and important in terms of system selection, to our best knowledge, no studies have compared these three tools as of yet.…”

“…Here, we addressed these issues by comparing the performance of three AU detection systems: FaceReader, OpenFace [ 6 , 7 ], and AFAR [ 8 , 9 ]. We used a cross-domain database such as in-the-wild data from YouTube videos, posed facial movements expressing emotional states and social facial expressions during conversations.…”