Modeling naturalistic face processing in humans with deep convolutional neural networks

Jiahui, Guo; Feilong, Ma; Castello, Matteo Visconti di Oleggio; Nastase, Samuel A.; Haxby, James V.; Gobbini, M. Ida

doi:10.1101/2021.11.17.469009

Cited by 9 publications

(5 citation statements)

References 117 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A more detailed investigation of layerwise encoding performance revealed a log-linear relationship where peak encoding performance tends to occur in relatively earlier layers as both model size and expressivity increase (Mischler et al, 2024). This is an unexpected extension of prior work on both language (Caucheteux & King, 2022;Kumar et al, 2022;Toneva & Wehbe, 2019) and vision (Jiahui et al, 2023), where peak encoding performance was found at late-intermediate layers. Moreover, we observed variations in best relative layers across different brain regions, corresponding to a language processing hierarchy.…”

Section: Discussionsupporting

confidence: 70%

Scale matters: Large language models with billions (rather than millions) of parameters better match neural representations of natural language

Hong,

Wang,

Zada

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

Recent research has used large language models (LLMs) to study the neural basis of naturalistic language processing in the human brain. LLMs have rapidly grown in complexity, leading to improved language processing capabilities. However, neuroscience researchers haven’t kept up with the quick progress in LLM development. Here, we utilized several families of transformer-based LLMs to investigate the relationship between model size and their ability to capture linguistic information in the human brain. Crucially, a subset of LLMs were trained on a fixed training set, enabling us to dissociate model size from architecture and training set size. We used electrocorticography (ECoG) to measure neural activity in epilepsy patients while they listened to a 30-minute naturalistic audio story. We fit electrode-wise encoding models using contextual embeddings extracted from each hidden layer of the LLMs to predict word-level neural signals. In line with prior work, we found that larger LLMs better capture the structure of natural language and better predict neural activity. We also found a log-linear relationship where the encoding performance peaks in relatively earlier layers as model size increases. We also observed variations in the best-performing layer across different brain regions, corresponding to an organized language processing hierarchy.

show abstract

Section: Discussionsupporting

confidence: 70%

Scale matters: Large language models with billions (rather than millions) of parameters better match neural representations of natural language

Hong,

Wang,

Zada

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…For example, object-trained and face-trained VGG models learn distinctly different feature detectors ( 7 ), yet explain a similar amount of variance in human face dissimilarity judgments. Object-trained and face-trained VGG models have previously been found to explain a similar amount of variance in human inferior temporal cortex ( 55 ) and in face-selective visual cortex ( 56 ), and object-trained VGG captured variance in early MEG responses ( 57 ). The face space within a face-trained DNN organizes faces differently than they are arranged in the BFM’s principal components, for example, clustering low-quality images at the “origin” of the space, eliciting lower activity from all learned features ( 42 ).…”

Section: Discussionmentioning

confidence: 91%

Face dissimilarity judgments are predicted by representational distance in morphable and image-computable models

Jozwik

O’Keeffe

Storrs

et al. 2022

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Human vision is attuned to the subtle differences between individual faces. Yet we lack a quantitative way of predicting how similar two face images look and whether they appear to show the same person. Principal component–based three-dimensional (3D) morphable models are widely used to generate stimuli in face perception research. These models capture the distribution of real human faces in terms of dimensions of physical shape and texture. How well does a “face space” based on these dimensions capture the similarity relationships humans perceive among faces? To answer this, we designed a behavioral task to collect dissimilarity and same/different identity judgments for 232 pairs of realistic faces. Stimuli sampled geometric relationships in a face space derived from principal components of 3D shape and texture (Basel face model [BFM]). We then compared a wide range of models in their ability to predict the data, including the BFM from which faces were generated, an active appearance model derived from face photographs, and image-computable models of visual perception. Euclidean distance in the BFM explained both dissimilarity and identity judgments surprisingly well. In a comparison against 16 diverse models, BFM distance was competitive with representational distances in state-of-the-art deep neural networks (DNNs), including novel DNNs trained on BFM synthetic identities or BFM latents. Models capturing the distribution of face shape and texture across individuals are not only useful tools for stimulus generation. They also capture important information about how faces are perceived, suggesting that human face representations are tuned to the statistical distribution of faces.

show abstract

“…Humans may be sensitive to dynamic information evolving across frames that LLaMA does not have access to; similar divergence between static and dynamic stimuli has been found in face recognition networks. 71 However, it is not clear how GPT-4 exactly processes GIF-or image-based input. Being likely trained on text and being based on the GPT architecture, it is unlikely that GPT-4 can meaningfully make use of temporal dynamics in the same way as, e.g.…”

Section: Differences Between Modelsmentioning

confidence: 99%

Multimodality and Attention Increase Alignment in Natural Language Prediction Between Humans and Computational Models

kewenig,

Lampinen,

Nastase

et al. 2024

Preprint

View full text Add to dashboard Cite

The potential of multimodal generative artificial intelligence (mAI) to replicate human grounded language understanding, including the pragmatic, context-rich aspects of communication, remains to be clarified. Humans are known to use salient multimodal features, such as visual cues, to facilitate the processing of upcoming words. Correspondingly, multimodal computational models can integrate visual and linguistic data using a visual attention mechanism to assign next-word probabilities. To test whether these processes align, we tasked both human participants (N = 200) as well as several state-of-the-art computational models with evaluating the predictability of forthcoming words after viewing short audio-only or audio-visual clips with speech. During the task, the model’s attention weights were recorded and human attention was indexed via eye tracking. Results show that predictability estimates from humans aligned more closely with scores generated from multimodal models vs. their unimodal counterparts. Furthermore, including an attention mechanism doubled alignment with human judgments when visual and linguistic context facilitated predictions. In these cases, the model’s attention patches and human eye tracking significantly overlapped. Our results indicate that improved modeling of naturalistic language processing in mAI does not merely depend on training diet but can be driven by multimodality in combination with attention-based architectures. Humans and computational models alike can leverage the predictive constraints of multimodal information by attending to relevant features in the input.

show abstract

Modeling naturalistic face processing in humans with deep convolutional neural networks

Cited by 9 publications

References 117 publications

Scale matters: Large language models with billions (rather than millions) of parameters better match neural representations of natural language

Scale matters: Large language models with billions (rather than millions) of parameters better match neural representations of natural language

Face dissimilarity judgments are predicted by representational distance in morphable and image-computable models

Multimodality and Attention Increase Alignment in Natural Language Prediction Between Humans and Computational Models

Contact Info

Product

Resources

About