Assessing Automated Facial Action Unit Detection Systems for Analyzing Cross-Domain Facial Expression Databases

Abstract

In the field of affective computing, achieving accurate automatic detection of facial movements is an important issue, and great progress has already been made. However, a systematic evaluation of systems that now have access to the dynamic facial database remains an unmet need. This study compared the performance of three systems (FaceReader, OpenFace, AFARtoolbox) that detect each facial movement corresponding to an action unit (AU) derived from the Facial Action Coding System. All machines could detect the presence of AUs from the dynamic facial database at a level above chance. Moreover, OpenFace and AFAR provided higher area under the receiver operating characteristic curve values compared to FaceReader. In addition, several confusion biases of facial components (e.g., AU12 and AU14) were observed to be related to each automated AU detection system and the static mode was superior to dynamic mode for analyzing the posed facial database. These findings demonstrate the features of prediction patterns for each system and provide guidance for research on facial expressions.

Viewpoint Robustness of Automated Facial Action Unit Detection Systems

“…OpenFace had the best AU detection performance among the systems evaluated in this study, consistent with the previous findings pertaining to dynamic facial expressions [19]. Among the pre-trained models, OpenFace had the best AU detection performance based on facial expression images.…”

“…FaceReader provided inaccurate results at 45 • . The poor performance of FaceReader for recognizing dynamic facial expressions reported in a previous study [19] may be due to its vulnerability to angle changes. In this study, the AUC values were significantly lower at 45 • , especially for AUs 1 (inner brow raised), 2 (outer brow raised), 5 (upper lid raised), 25 (lips parted), and 26 (jaw dropped).…”

“…Area under the curve (AUC) values are superior to other indices (such as F1 scores) for comparison of the performance of automated facial action detection systems and were thus used here [19,44].…”

“…Commercial tools are expensive, and open-source software often lacks user-friendly graphical user interfaces. To evaluate the software available to non-programmers with the best graphical user interface and performance for facial action detection, Namba et al [19] compared the performances of three automatic facial action detection systems (AFAR toolbox, OpenFace, and FaceReader) using several dynamic facial databases, including one derived from YouTube. OpenFace [16] was able to detect facial expressions more accurately than the other automatic facial action detection systems, including commercial software.…”

An Android for Emotional Interaction: Spatiotemporal Validation of Its Facial Expressions

“…Moreover, like several other advanced androids (e.g., Glas et al, 2016;Ishi et al, 2017Ishi et al, , 2019, Nikola has the ability to talk with prosody, which can facilitate multimodal emotional interactions (Paulmann and Pell, 2011). Androids can also utilize advanced artificial intelligence (for reviews, see Krumhuber et al, 2021;Namba et al, 2021) to sense and analyze human facial expressions. We expect that androids will be a valuable tool in future psychological research on human emotional interaction.…”