Mapping the emotional face. How individual face parts contribute to successful emotion recognition

Abstract

Which facial features allow human observers to successfully recognize expressions of emotion? While the eyes and mouth have been frequently shown to be of high importance, research on facial action units has made more precise predictions about the areas involved in displaying each emotion. The present research investigated on a fine-grained level, which physical features are most relied on when decoding facial expressions. In the experiment, individual faces expressing the basic emotions according to Ekman were hidden behind a mask of 48 tiles, which was sequentially uncovered. Participants were instructed to stop the sequence as soon as they recognized the facial expression and assign it the correct label. For each part of the face, its contribution to successful recognition was computed, allowing to visualize the importance of different face areas for each expression. Overall, observers were mostly relying on the eye and mouth regions when successfully recognizing an emotion. Furthermore, the difference in the importance of eyes and mouth allowed to group the expressions in a continuous space, ranging from sadness and fear (reliance on the eyes) to disgust and happiness (mouth). The face parts with highest diagnostic value for expression identification were typically located in areas corresponding to action units from the facial action coding system. A similarity analysis of the usefulness of different face parts for expression recognition demonstrated that faces cluster according to the emotion they express, rather than by low-level physical features. Also, expressions relying more on the eyes or mouth region were in close proximity in the constructed similarity space. These analyses help to better understand how human observers process expressions of emotion, by delineating the mapping from facial features to psychological representation.

Dynamic representations of faces in the human ventral visual stream link visual features to behaviour

“…Furthermore, ventral stream areas encoded facial features prior to facial configuration when faces were presented for 150 ms. This adds to evidence suggesting that emotional face perception is supported by the processing of diagnostic features, such as the eyes and mouth (Wegrzyn, Vogt, Kireclioglu, Schneider, & Kissler, 2017). What is more, configural representations explain behaviour and overlap with behavioural representations, suggesting that it is face configuration that drives expression-selective responses in ventral stream areas and guides behaviour.…”

Visual scanning during emotion recognition in long‐term recovered anorexia nervosa: An eye‐tracking study

“…A potential mechanism might be deviant visual scanning behavior. For example, the role of attending to the eyes varies for the recognition of different emotions: while the mouth region contains critical information for identifying happiness, the recognition of fear and anger relies mainly on cues from the eye region (Smith, Cottrell, Gosselin, & Schyns, 2005;Wegrzyn, Vogt, Kireclioglu, Schneider, & Kissler, 2017). FER deficits for negative emotions might therefore originate in reduced attention to the eye region.…”

Causal Inference in Nonverbal Dyadic Communication with Relevant Interval Selection and Granger Causality

“…In this paper, we employed GC together with a relevant interval selection approach on synthetic and nonverbal communication data obtained from an experimental setup. Based on the results of Wegrzyn et al (Wegrzyn et al, 2017) we designed our own emotional facial features, capable of capturing emotions even when strong distinct emotions are not visible. Our facial expressions are composed of facial action units, which can be detected by real-time, state of the art computer vision tools.…”

“…Wegrzyn et al (Wegrzyn et al, 2017) studied the relevance of facial areas for emotion classification and found differences in the importance of the eye and mouth regions. Facial AUs can be divided into upper and lower AUs (Cohn et al, 2007).…”