Viewpoint Robustness of Automated Facial Action Unit Detection Systems

Namba, Shushi; Sato, Wataru; Yoshikawa, Sakiko

doi:10.3390/app112311171

Cited by 7 publications

(6 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…FaceReader 9.0 also provided evidence that the side of the face with the electrode attachment for ZM (participant's left face, which was the right side in the video recordings) demonstrated lower sensitivity, positive predictive value, and F1 score than the side without the electrode. This showed that in real-life applications, the automated FACS would not be robust against the presence of facial accessories or scars in addition to the effect of ever-changing viewpoints [24]. FaceReader, which could evaluate bilateral AUs separately, seems more useful against partial visual blockade.…”

Section: Discussionmentioning

confidence: 99%

“…An XGBoost classifier, Feat-XGB [67], was used as the AU detector, which used PCA-reduced HOG features for AU predictions, as with OpenFace [54]. It was trained for 20 AUs (1, 2, 4, 5, 6, 7, 9, 10, 12, 14, 15, 17, 18, 20, 23, 24, 25, 26, 28, 43) using BP4D [57], BP4D+ [68], DISFA [55], DISFA+ [69], CK+ [70], JAFFE [71], Shoulder Pain [58], and EmotioNet [72] and validated using WIDER FACE [51], 300W [73], NAMBA [24], and BIWI-Kinect [74]. The average F1 score was 0.54 (AU4 = 0.64 and AU12 = 0.83) [42].…”

Section: Py-featmentioning

confidence: 99%

“…With advances in computer vision and machine learning, the FACS has been incorporated by several automatic facial behavior software, and studies have investigated the validity of such tools. Still, most of these studies have either compared automated AUcoding performances with human coder performances [21][22][23][24][25] or tested the performance of emotion recognition from labeled facial expression datasets [22,26].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Electromyographic Validation of Spontaneous Facial Mimicry Detection Using Automated Facial Action Coding

Hsu,

Sato

2023

Sensors

Self Cite

View full text Add to dashboard Cite

Although electromyography (EMG) remains the standard, researchers have begun using automated facial action coding system (FACS) software to evaluate spontaneous facial mimicry despite the lack of evidence of its validity. Using the facial EMG of the zygomaticus major (ZM) as a standard, we confirmed the detection of spontaneous facial mimicry in action unit 12 (AU12, lip corner puller) via an automated FACS. Participants were alternately presented with real-time model performance and prerecorded videos of dynamic facial expressions, while simultaneous ZM signal and frontal facial videos were acquired. Facial videos were estimated for AU12 using FaceReader, Py-Feat, and OpenFace. The automated FACS is less sensitive and less accurate than facial EMG, but AU12 mimicking responses were significantly correlated with ZM responses. All three software programs detected enhanced facial mimicry by live performances. The AU12 time series showed a roughly 100 to 300 ms latency relative to the ZM. Our results suggested that while the automated FACS could not replace facial EMG in mimicry detection, it could serve a purpose for large effect sizes. Researchers should be cautious with the automated FACS outputs, especially when studying clinical populations. In addition, developers should consider the EMG validation of AU estimation as a benchmark.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Py-featmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Electromyographic Validation of Spontaneous Facial Mimicry Detection Using Automated Facial Action Coding

Hsu,

Sato

2023

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…The Facial Action Coding System considers AUs as having the ability to describe all facial movements anatomically [28]. While OpenFace does not guarantee the same performance that manual facial coding does, there was sufficient biserial correlation (r = .80) between OpenFace and expert FACS coders' performances to static frontal facial images of Japanese persons [43]. OpenFace can detect 18 AUs: 1 (inner brow raiser), 2 (outer brow raiser), 4 (brow lowerer), 5 (upper lid raiser), 6 (cheek raiser), 7 (lid tightener), 9 (nose wrinkler), 10 (upper lip raiser), 12 (lip corner puller), 14 (dimpler), 15 (lip corner depressor), 17 (chin raiser), 20 (lip stretcher), 23 (lip tightener), 25 (lips parts), 26 (jaw drop), 28 (lip suck), and 45 (blink).…”

Section: Discussionmentioning

confidence: 99%

The spatio-temporal features of perceived-as-genuine and deliberate expressions

2022

Self Cite

View full text Add to dashboard Cite

Reading the genuineness of facial expressions is important for increasing the credibility of information conveyed by faces. However, it remains unclear which spatio-temporal characteristics of facial movements serve as critical cues to the perceived genuineness of facial expressions. This study focused on observable spatio-temporal differences between perceived-as-genuine and deliberate expressions of happiness and anger expressions. In this experiment, 89 Japanese participants were asked to judge the perceived genuineness of faces in videos showing happiness or anger expressions. To identify diagnostic facial cues to the perceived genuineness of the facial expressions, we analyzed a total of 128 face videos using an automated facial action detection system; thereby, moment-to-moment activations in facial action units were annotated, and nonnegative matrix factorization extracted sparse and meaningful components from all action units data. The results showed that genuineness judgments reduced when more spatial patterns were observed in facial expressions. As for the temporal features, the perceived-as-deliberate expressions of happiness generally had faster onsets to the peak than the perceived-as-genuine expressions of happiness. Moreover, opening the mouth negatively contributed to the perceived-as-genuine expressions, irrespective of the type of facial expressions. These findings provide the first evidence for dynamic facial cues to the perceived genuineness of happiness and anger expressions.

show abstract

“…Previous studies found that OpenFace performed AU detection at higher-than-chance levels, for posed datasets as well as videos collected "in the wild" from YouTube (Namba et al, 2021a). However, at least for still images, the accuracy drops when the face is angled at 45 • , but still higher than chance (Namba et al, 2021b). Examples of the four Action Units are provided in Figure 2.…”

Section: Eyebrow Movement Analysis-openface 20mentioning

confidence: 93%

Multimodal Prominence Marking in Semi-Spontaneous YouTube Monologs: The Interaction of Intonation and Eyebrow Movements

Berger

Zellers

2022

Front. Commun.

View full text Add to dashboard Cite

Both facial expressions like eyebrow movements and prosodic characteristics like pitch height and the position of the pitch accent relative to the prominent syllable play an important role in prominence marking, which in turn is used by YouTubers and other public speakers alike to engage their audience. We analyze the co-occurrence of eyebrow movements and pitch accents of differing prominence levels produced by a sample of English-speaking YouTubers, and the height and position of those pitch accents. We annotated pitch accents, and analyzed videos using OpenFace 2.0, using three different eyebrow movement measures: eyebrow landmarks, distance between eyebrow and eye landmarks, and Action Units (facial expressions). We found that pitch height and eyebrow movements are positively correlated for at least some of the measures for all speakers. We also found that it is more likely for a medial or late pitch accent to arise at the time of an Action Unit connected to the eyebrows than an early accent, while there was no effect of the prominence level. However, we did not find evidence for a close temporal coordination of pitch movement and eyebrow movement. Finally, we found speaker-individual differences in the use of eyebrow movements and pitch accents, but also some slight indications of an effect for geographical origin of the speaker.

show abstract

Viewpoint Robustness of Automated Facial Action Unit Detection Systems

Cited by 7 publications

References 50 publications

Electromyographic Validation of Spontaneous Facial Mimicry Detection Using Automated Facial Action Coding

Electromyographic Validation of Spontaneous Facial Mimicry Detection Using Automated Facial Action Coding

The spatio-temporal features of perceived-as-genuine and deliberate expressions

Multimodal Prominence Marking in Semi-Spontaneous YouTube Monologs: The Interaction of Intonation and Eyebrow Movements

Contact Info

Product

Resources

About