Explaining face representation in the primate brain using different computational models

Abstract

Understanding how the brain represents the identity of complex objects is a central challenge of visual neuroscience. The principles governing object processing have been extensively studied in the macaque face patch system, a sub-network of inferotemporal (IT) cortex specialized for face processing (Tsao et al., 2006). A previous study reported that single face patch neurons encode axes of a generative model called the "active appearance" model (Chang and Tsao, 2017), which transforms 50-d feature vectors separately representing facial shape and facial texture into facial images (Cootes et al., 2001;Edwards et al., 1998). However, it remains unclear whether this model constitutes the best model for explaining face cell responses. Here, we recorded responses of cells in the most anterior face patch AM to a large set of real face images, and compared a large number of models for explaining neural responses. We found that the active appearance model better explained responses than any other model except CORnet-Z, a feedforward deep neural network trained on general object classification to classify non-face images, whose performance it tied on some face image sets and exceeded on others. Surprisingly, deep neural networks trained specifically on facial identification did not explain neural responses well. A major reason is that units in the network, unlike neurons, are less modulated by face-related factors unrelated to facial identification such as illumination.

Face dissimilarity judgements are predicted by representational distance in morphable and image-computable models

“…Another possibility is that human face dissimilarity judgements are based on general-purpose descriptions of high-level image structure, which are not specific to faces. Consistent with our behavioural results, VGG trained on faces was not beneficial in explaining neural recordings of faces (50). These results are consistent with our finding that a general object recognition model seems to be sufficient to develop features that are diagnostic of face dissimilarity.…”

A connectivity-constrained computational account of topographic organization in primate high-level visual cortex

“…Thus, it is intriguing to note that whereas face selectivity might emerge without substantial experience at the individual level or any specialized mechanisms (see also Cowell and Cottrell, 2013), such extensive experience appears to be necessary to induce a representational space capable of supporting rapid, expert-level individual face recognition (see also Young and Burton, 2018;Sunday and Gauthier, 2018;Blauch et al, 2021b;Yovel and Abudarham, 2021;Young, 2021). The fact that a DCNN model trained without individual-level face experience is able to reproduce many of the hallmarks of face selectivity in primate cortex (Lee et al, 2020), and may provide a superior fit to one trained only with individual-level face experience (Chang et al, 2020), suggests that non-human primates may not be the optimal model system for studying the neural basis of human expert individual face recognition (Rossion and Taubert, 2017;Rossion, 2019). Given that many aspects of the non-human primate visual system make it an excellent model system for studying visual functional organization more generally, perhaps caution is warranted when human and non-human primates do not perform a given task with equivalent proficiency.…”

Grounding deep neural network predictions of human categorization behavior in understandable functional features: The case of face identity

“…33,[54][55][56][57][58][59] In this context, the exact structure of the information represented in the human brain remains an empirical question. The veridical representation implied by computer graphics models 53,60,61 is one hypothesis. Other specific ideas about face, object, and scene representations must and will be tested with different designs of generative models, including DNNs (e.g., VanRullen and Reddy, [62][63][64] Bashivan et al, [62][63][64] Ponce et al [62][63][64] ).…”