The contribution of different facial regions to the recognition of conversational expressions

Abstract

The human face is an important and complex communication channel. Humans can, however, easily read in a face not only identity information but also facial expressions with high accuracy. Here, we present the results of four psychophysical experiments in which we systematically manipulated certain facial areas in video sequences of nine conversational expressions to investigate recognition performance and its dependency on the motions of different facial parts. The results help to demonstrate what information is perceptually necessary and sufficient to recognize the different facial expressions. Subsequent analyses of the facial movements and correlation with recognition performance show that, for some expressions, one individual facial region can represent the whole expression. In other cases, the interaction of more than one facial area is needed to clarify the expression. The full set of results is used to develop a systematic description of the roles of different facial parts in the visual perception of conversational facial expressions.

Neural processing of fearful and happy facial expressions during emotion-relevant and emotion-irrelevant tasks: A fixation-to-feature approach

“…Some features characterize particular facial expressions better than others, like the smiling mouth for happy faces and the wide open eyes for fearful faces (e.g., Kohler et al, 2004;Leppänen & Hietanen, 2007;Nusseck, Cunningham, Wallraven, & Bülthoff, 2008;Smith, Cottrell, Gosselin, & Schyns, 2005). Behavioural research presenting face parts (e.g., Calder, Young, Keane, & Dean,2000) or using response classification techniques such as Bubbles (e.g., Blais, Roy, Fiset, Arguin, & Gosselin, 2012;Smith et al, 2005) has highlighted the importance of these so-called "diagnostic features" for the discrimination and categorization of these facial emotions.…”

The Face Speaks: Contextual and Temporal Sensitivity to Backchannel Responses

“…In future work, we will be constructing full active-appearance models of the speakers and listeners in the database. These will be used to create video sequences modified to freeze certain facial parts (e.g., [7]), to warp the timing of the backchannel responses, etc. With these modified sequences, we can conduct more detailed experiments on the sensitivity to physical changes for facial expressions in conversational contexts.…”

“…More specifically, we will present two experiments that aim to test the contextual and temporal sensitivity for processing of facial feedback signals. These experiments are intended to provide important contributions towards full spatio-temporal modeling of the non-verbal facial (and head-related) information channel in conversations, complementing previous research on conversational facial expressions (e.g., [7]). These kinds of models will be indispensable for the creation of virtual agents or artificial listeners that can display human-like listener behaviors [6,14].…”

“…In recent years there has been an effort to examine other expressions that occur in conversations with a higher frequency (such as thinking, agreeing, being confused, being bored, etc. ): [4][5][6][7][8]. Conversational expressions are not limited to movements of facial muscles, they also include global head motion and orientation (e.g.…”

“…to indicate the addressee of a question). Which regions of the face are necessary and sufficient for expression recognition was investigated in [7]. They showed that the motion of different face regions contribute a varying amount to recognition performance.…”