Morphing between expressions dissociates continuous from categorical representations of facial expression in the human brain

Abstract

Whether the brain represents facial expressions as perceptual continua or as emotion categories remains controversial. Here, we measured the neural response to morphed images to directly address how facial expressions of emotion are represented in the brain. We found that face-selective regions in the posterior superior temporal sulcus and the amygdala responded selectively to changes in facial expression, independent of changes in identity. We then asked whether the responses in these regions reflected categorical or continuous neural representations of facial expression. Participants viewed images from continua generated by morphing between faces posing different expressions such that the expression could be the same, could involve a physical change but convey the same emotion, or could differ by the same physical amount but be perceived as two different emotions. We found that the posterior superior temporal sulcus was equally sensitive to all changes in facial expression, consistent with a continuous representation. In contrast, the amygdala was only sensitive to changes in expression that altered the perceived emotion, demonstrating a more categorical representation. These results offer a resolution to the controversy about how facial expression is processed in the brain by showing that both continuous and categorical representations underlie our ability to extract this important social cue.face | temporal lobe | fMRI T he ability to visually encode changes in facial musculature that reflect emotional state is essential for effective social communication (1). Models of face processing have proposed either that the perception of facial expression is dependent on a continuous neural representation of gradations in expression along critical dimensions (2, 3) or that it is based primarily on the assignment of expressions to discrete emotion categories (4, 5).Although they are usually treated as incompatible opposites, there is evidence consistent with both the continuous and categorical approaches to facial expression perception (6). The categorical perspective is based on the idea that distinct neural or cognitive states underpin a set of basic facial expressions (5). Evidence for categorical perception of expression is shown by the consistency with which the basic emotions are recognized (7) and the greater sensitivity to changes in facial expression that alter the perceived emotion (8, 9). In contrast, continuous or dimensional models are better able to explain the systematic confusions that occur when labeling facial expressions (2) and can account for variation in the way that basic emotions are expressed (10) and the fact that we are readily able to perceive differences in intensity of a given emotional expression (11,12).The aim of this study is to ask whether primarily categorical or continuous representations of facial expression are used in different regions of the human brain. Neuroimaging studies have identified a number of face-selective regions that are involved in the perception of facial express...

Facial identity and facial expression are initially integrated at visual perceptual stages of face processing

“…In line with the classic account proposed by Haxby et al (2000), the FFA has been shown to be sensitive to repetitions of face identity (Rotshtein, Henson, Treves, Driver, & Dolan, 2005;Winston, Henson, Fine-Goulden, & Dolan, 2004;Yovel & Kanwisher, 2005), while the STS was found to be sensitive to dynamic aspects of face perception such as repetitions of expression (Andrews & Ewbank, 2004;Ganel, Valyear, Goshen-Gottstein, & Goodale, 2005;Harris, Young, & Andrews, 2012;Winston, Henson, Fine-Goulden, & Dolan, 2004). A recent fMRI adaptation study by Harris, Young, and Andrews (2014) has suggested that brain regions involved in the recognition of facial identity or expression are differentially sensitive to surface-based versus edge-based visual signals.…”

“…Activity in the FFA has been shown to be larger in response to expressive versus neutral faces (Furl, Henson, Friston, & Calder, 2013;Vuilleumier, Armony, Driver, & Dolan, 2001), and adaptation effects by repetitions of facial expressions have been found in the FFA (Ganel et al, 2005;Harris et al, 2012;Fox, Moon, Iaria, & Barton, 2009). Furthermore, there is also fMRI evidence that the STS is sensitive to repetitions of face identity (Fox et al, 2009).…”

MEG Adaptation Resolves the Spatiotemporal Characteristics of Face-Sensitive Brain Responses

“…These structures form the core components of an influential neurocognitive model of the human face-processing system (Haxby et al, 2000) that has received broad support in terms of its general claims (GrillSpector et al, 2004;Winston et al, 2004;Fox et al, 2009), although more recent work continues to fine-tune our understanding of the component nodes of this network and their functional sensitivities (Said et al, 2010;van den Hurk et al, 2011;Harris et al, 2012;Mende-Siedlecki et al, 2013;Baseler et al, 2014).…”

Perceiving expressions of emotion: What evidence could bear on questions about perceptual experience of mental states?

“…Answering this question is complicated by the fact that many parts of the brain are involved (Adolphs 2002;Vuilleumier & Pourtois 2007). ere is evidence that both the amygdala (Harris et al 2012(Harris et al , 2014 and also some cortical structures (Bay & Taylor 2003) respond categorically to expressions of emotion; and that intervening in the operations of the somatosensory cortex can impair categorisation (Pitcher et al 2008; see also Banissy et al 2011). To my knowledge, so far it is only for happy and fearful stimuli that we have direct evidence from both neurophysiological (Campanella et al 2002) and behavioural measures (Williams et al 2005) of categorisation occurring in perceptual processing.…”