Adjudicating between face-coding models with individual-face fMRI responses

Abstract

The perceptual representation of individual faces is often explained with reference to a norm-based face space. In such spaces, individuals are encoded as vectors where identity is primarily conveyed by direction and distinctiveness by eccentricity. Here we measured human fMRI responses and psychophysical similarity judgments of individual face exemplars, which were generated as realistic 3D animations using a computer-graphics model. We developed and evaluated multiple neurobiologically plausible computational models, each of which predicts a representational distance matrix and a regional-mean activation profile for 24 face stimuli. In the fusiform face area, a face-space coding model with sigmoidal ramp tuning provided a better account of the data than one based on exemplar tuning. However, an image-processing model with weighted banks of Gabor filters performed similarly. Accounting for the data required the inclusion of a measurement-level population averaging mechanism that approximates how fMRI voxels locally average distinct neuronal tunings. Our study demonstrates the importance of comparing multiple models and of modeling the measurement process in computational neuroimaging.Author SummaryHumans recognize conspecifics by their faces. Understanding how faces are recognized is an open computational problem with relevance to theories of perception, social cognition, and the engineering of computer vision systems. Here we measured brain activity with functional MRI while human participants viewed individual faces. We developed multiple computational models inspired by known response preferences of single neurons in the primate visual cortex. We then compared these neuronal models to patterns of brain activity corresponding to individual faces. The data were consistent with a model where neurons respond to directions in a high-dimensional space of faces. It also proved essential to model how functional MRI voxels locally average the responses of tens of thousands of neurons. The study highlights the challenges in adjudicating between alternative computational theories of visual information processing.

FFA and OFA encode distinct types of face identity information

“…In contrast, the representational geometries of faces in the right OFA were most associated with lowlevel image-based properties, such as pixel similarity and features extracted with Gabor filters that simulate functioning of early visual cortex. While previous studies had shown that low-level properties of images extracted with Gabor filters were associated with representational distances of faces in right FFA (Carlin & Kriegeskorte, 2017;Weibert et al, 2018), our results suggest that representations in right FFA use more complex combinations of stimulus-based features and relate to higher-level perceived and social properties. These results inform existing neurocognitive models of face processing (Haxby et al, 2000;Duchaine & Yovel, 2015) by shedding light on the muchdebated computations of face-responsive regions, and providing new evidence to support a hierarchical organisation of these regions from the processing of low-level image-computable properties in the OFA to higher-level visual features and social information in the FFA.…”

“…These results thus suggest that the OFA and FFA both represent complex configurations of image-based information and not face identity per se. Some studies have also shown that even lower-level stimulus-based properties of face images, such as those computed by Gabor filters, explain significant variance in the representational geometries in the FFA (Carlin & Kriegeskorte, 2017) as well as OFA and pSTS (Weibert et al, 2018). On the other hand, other studies have shown that more high-level information, such as biographical information and social context, affects the similarity of response patterns to different faces in the FFA (Verosky et al, 2013;Collins et al, 2016).…”

Changes within neural population codes can be inferred from psychophysical threshold studies

“…Fourth, we assumed a bellshaped tuning function, which as far as we know is the only type of function used in research applying inverted encoding modeling [e.g., 9,38,31,15,13,11,12,10]. For other stimulus dimensions, such as those characterized by monotonically increasing or decreasing tuning functions (e.g., face dimensions; [44,5,45,43]), new simulations would be required. Finally, we assumed a homogeneous encoding population for the baseline condition.…”

“…For orientation stimuli, we chose to model channels as symmetrical Gaussians with Poisson noise based on previous literature, but there are other ways to characterize tuning functions that may be more appropriate for other types of stimuli. For example, face dimensions are often modeled using asymmetric sigmoidal tuning functions [44,5,45,43]. Also, another common assumption for neural channel noise is that it follows a Gaussian distribution [e.g., 6, 7], a common assumption implicit in inverted encoding modeling analyses of neuroimaging data [see 8,14].…”

Individual-subject functional localization increases univariate activation but not multivariate pattern discriminability in the ‘multiple-demand’ frontoparietal network

“…To ensure that our decoding results did not depend on the choice of classifier, we repeated the MVPA using a representational similarity analysis (RSA) approach, with linear discriminant contrast (LDC) as a measure of dissimilarity between rule patterns (Carlin & Kriegeskorte, 2017;Nili et al, 2014). Cross-validated Mahalanobis distances were calculated for all 15 pairwise rule combinations and averaged to get within-and between-category rule pairs, for each ROI and participant, using all the voxels in the ROI and subject-specific ROIs defined using the different localizer tasks.…”