Human perception and biosignal-based identification of posed and spontaneous smiles

Abstract

Facial expressions are behavioural cues that represent an affective state. Because of this, they are an unobtrusive alternative to affective self-report. The perceptual identification of facial expressions can be performed automatically with technological assistance. Once the facial expressions have been identified, the interpretation is usually left to a field expert. However, facial expressions do not always represent the felt affect; they can also be a communication tool. Therefore, facial expression measurements are prone to the same biases as self-report. Hence, the automatic measurement of human affect should also make inferences on the nature of the facial expressions instead of describing facial movements only. We present two experiments designed to assess whether such automated inferential judgment could be advantageous. In particular, we investigated the differences between posed and spontaneous smiles. The aim of the first experiment was to elicit both types of expressions. In contrast to other studies, the temporal dynamics of the elicited posed expression were not constrained by the eliciting instruction. Electromyography (EMG) was used to automatically discriminate between them. Spontaneous smiles were found to differ from posed smiles in magnitude, onset time, and onset and offset speed independently of the producer’s ethnicity. Agreement between the expression type and EMG-based automatic detection reached 94% accuracy. Finally, measurements of the agreement between human video coders showed that although agreement on perceptual labels is fairly good, the agreement worsens with inferential labels. A second experiment confirmed that a layperson’s accuracy as regards distinguishing posed from spontaneous smiles is poor. Therefore, the automatic identification of inferential labels would be beneficial in terms of affective assessments and further research on this topic.

Smiles as a Signal of Prosocial Behaviors Toward the Robot in the Therapeutic Setting for Children With Autism Spectrum Disorder

“…This device can detect the contractions of facial muscles related to smiling-the orbicularis oculi and zygomaticus major. These facial muscle areas have been researched with EMG sensors to measure specific smiles that show spontaneous and positive emotions (Frank et al, 1993;Mauss and Robinson, 2009;Johnson et al, 2010;Perusquía-Hernández et al, 2019). Compared to other physiological sensors, such as electroencephalography and functional MRI, facial EMG can be attached directly to the facial muscles involved in smiling (Maria et al, 2019).…”

Assessing Automated Facial Action Unit Detection Systems for Analyzing Cross-Domain Facial Expression Databases

“…First, system accuracy when targeting one facial stimulus does not guarantee accuracy when targeting another facial stimulus, i.e., their generalizability across various domains is unproven. The interest in affective studies has shifted from posed and static facial images to more spontaneous and dynamic facial movements [ 12 , 13 , 14 , 15 ]. In this situation, the most important question for affective computing systems is “to what extent does the automatic AU detection system generalize to other data in the various domains.” To the best of our knowledge, how well open-source AU detection systems and even commercial AU detection software can perform with spontaneous and dynamic facial movements compared to human FACS coding remains unknown.…”

Are people happy when they smile? Affective assessments based on automatic smile genuineness identification

“…Moreover, posed or polite facial expressions might differ depending on cultural background (Thibault, Gosselin, Brunel, & Hess, 2009;Thibault, Levesque, Gosselin, & Hess, 2012) and racial match (Hourihan, Benjamin, & Liu, 2012;Meissner & Brigham, 2001). Only few studies targeted participants with mixed cultural and ethnic backgrounds (Cohn et al, 2004;Perusquía-Hernández, Ayabe-Kanamura, & Suzuki, 2019a). Future research should embrace diversity, as this is likely to play a factor in the generalisation of automatic recognition.…”

“…Despite the ambiguous definition of smile offsets and temporal units, it is possible to observe a familiar shape in several studies (Cohn & Schmidt, 2004;Jarlier et al, 2011;Mavadati et al, 2016;Perusquía-Hernández, Ayabe-Kanamura, & Suzuki, 2019a;Perusquía-Hernández et al, 2017b;Saito et al, 2020;Schmidt et al, 2006). Moreover, different studies have shown that when posing a smile for the camera, or under instruction, the decay of the smile is sharper than that of spontaneous smiles of enjoyment (Perusquía-Hernández, Ayabe-Kanamura, & Suzuki, 2019a;Perusquía-Hernández et al, 2017b;Saito et al, 2020). This might be the most distinct feature of posed smiles elicited in this manner, and it seems easy to distinguish given the high accuracy of several detection systems.…”