Explaining face representation in the primate brain using different computational models

Chang, Le; Egger, Bernhard; Vetter, Thomas; Tsao, Doris Y.

doi:10.1016/j.cub.2021.04.014

Cited by 35 publications

(35 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is worth noting that here we also explored other DNN models (Supplementary Fig. 5 ) and found that our findings could generalize to other DNNs (although as expected face-recognition performance was reduced in some DNNs that were not trained for face recognition), consistent with a previous report surveying a large class of DNN models for face representation 37 . Interestingly, a recent study has even shown that face-selective units can emerge from an untrained DNN 38 .…”

Section: Discussionsupporting

confidence: 90%

Face identity coding in the deep neural network and primate brain

et al. 2022

View full text Add to dashboard Cite

A central challenge in face perception research is to understand how neurons encode face identities. This challenge has not been met largely due to the lack of simultaneous access to the entire face processing neural network and the lack of a comprehensive multifaceted model capable of characterizing a large number of facial features. Here, we addressed this challenge by conducting in silico experiments using a pre-trained face recognition deep neural network (DNN) with a diverse array of stimuli. We identified a subset of DNN units selective to face identities, and these identity-selective units demonstrated generalized discriminability to novel faces. Visualization and manipulation of the network revealed the importance of identity-selective units in face recognition. Importantly, using our monkey and human single-neuron recordings, we directly compared the response of artificial units with real primate neurons to the same stimuli and found that artificial units shared a similar representation of facial features as primate neurons. We also observed a region-based feature coding mechanism in DNN units as in human neurons. Together, by directly linking between artificial and primate neural systems, our results shed light on how the primate brain performs face recognition tasks.

show abstract

Section: Discussionsupporting

confidence: 90%

Face identity coding in the deep neural network and primate brain

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Previous studies have successfully used DNN-based encoding models to characterize the tuning in category-selective visual areas and predict response patterns for a held-out test set. However, in these studies all image categories were represented in both the training and test set of the encoding model (Yamins et al , 2014; Güçlü and van Gerven, 2015; Kalfas, Kumar and Vogels, 2017; Murty et al , 2021), while some used only face images for both training and testing (Chang et al , 2021). For this reason, these studies cannot distinguish whether category selectivity in brain responses was determined by domain-specific features (e.g., face parts) that correlate well with DNN activations, or by domain-general image attributes that generalize across category boundaries.…”

Section: Discussionmentioning

confidence: 99%

The neural code for ‘face cells’ is not face specific

Vinken

Konkle

Livingstone

2022

Preprint

View full text Add to dashboard Cite

'Face cells' are visual neurons that selectively respond more to faces than other objects. Clustered together in inferotemporal cortex, they are thought to form a network of modules specialized in face processing by encoding face-specific features. Here we reveal that their category selectivity is instead captured by domain-general attributes. Analyzing neural responses in and around macaque face patches to hundreds of objects, we discovered graded tuning for non-face objects that was more predictive of face preference than was tuning for faces themselves. The relationship between category-level face selectivity and image-level non-face tuning was not predicted by color and simple shape properties, but by domain-general information encoded in deep neural networks trained on object classification. These findings contradict the long-standing assumption that face cells owe their category selectivity to face-specific features, challenging the prevailing idea that visual processing is carried out by discrete modules, each specialized in a semantically distinct domain.

show abstract

“…Studies investigating human face processing reported similarity to DCNNs, and, recently, a growing literature advocates the use of DCNNs as models to understand the neural basis of face processing in humans (Dobs et al, 2022; Grossman et al, 2019; Kuzovkin et al, 2018; Murty et al, 2021; Tsantani et al, 2021). However, correlations reported between DCNNs and neural representations of faces are often weak in humans (Kuzovkin et al, 2018; Tsantani et al, 2021), and a recent study has challenged how well DCNNs capture face representation in the macaque brain (Chang et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

Modeling naturalistic face processing in humans with deep convolutional neural networks

Jiahui

Feilong

Castello

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep convolutional neural networks (DCNNs) trained for face identification can rival and even exceed human-level performance. The relationships between internal representations learned by DCNNs and those of the primate face processing system are not well understood, especially in naturalistic settings. We developed the largest naturalistic dynamic face stimulus set in human neuroimaging research (700+ naturalistic video clips of unfamiliar faces) and used representational similarity analysis to investigate how well the representations learned by high-performing DCNNs match human brain representations across the entire distributed face processing system. DCNN representational geometries were strikingly consistent across diverse architectures and captured meaningful variance among faces. Similarly, representational geometries throughout the human face network were highly consistent across subjects. Nonetheless, correlations between DCNN and neural representations were very weak overall—DCNNs captured 3% of variance in the neural representational geometries at best. Intermediate DCNN layers better matched visual and face-selective cortices than the final fully-connected layers. Behavioral ratings of face similarity were highly correlated with intermediate layers of DCNNs, but also failed to capture representational geometry in the human brain. Our results suggest that the correspondence between intermediate DCNN layers and neural representations of naturalistic human face processing is weak at best, and diverges even further in the later fully-connected layers. This poor correspondence can be attributed, at least in part, to the dynamic and cognitive information that plays an essential role in human face processing but is not modeled by DCNNs. These mismatches indicate that current DCNNs have limited validity as in silico models of dynamic, naturalistic face processing in humans.

show abstract

Explaining face representation in the primate brain using different computational models

Cited by 35 publications

References 42 publications

Face identity coding in the deep neural network and primate brain

Face identity coding in the deep neural network and primate brain

The neural code for ‘face cells’ is not face specific

Modeling naturalistic face processing in humans with deep convolutional neural networks

Contact Info

Product

Resources

About