Spatiotemporal properties of the neural representation of conceptual content for words and pictures – an MEG study

Giari, Giuliano; Leonardelli, Elisa; Tao, Yuan Xiang; Machado, Mayara de Oliveira Silva; Fairhall, Scott L.

doi:10.1016/j.neuroimage.2020.116913

Cited by 25 publications

(20 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…19) = 1.99, p = 0.03, d = 0.46). We note that these mean values, while low, are completely consistent with prior literature on 2OI between RDMs produced from DNNs and MEG data (e.g., Giari et al, 2020) and the effect size at the group level (d = 0.46) suggested a substantial impact of pruning even in a domain with essentially low 2OI.…”

Section: Meg Resultssupporting

confidence: 89%

“…Similarity was defined as the correlation between the MEG activation vectors of images k and j over all 360 sensors. This is the regular procedure using MEG (e.g., Giari et al, 2020) as each MEG sensor reflects contributions of multiple activation sources in the brain, so that activity is not strongly localized topographically.…”

Section: Methods: Learning Pruning From Meg Datamentioning

confidence: 99%

See 1 more Smart Citation

Improved prediction of behavioral and neural similarity spaces using pruned DNNs

Tarigopula

Fairhall

Hasson

2021

Preprint

View full text Add to dashboard Cite

Deep Neural Networks (DNNs) have become an important tool for modeling brain and behaviour. One key area of interest has been to apply these networks to model human similarity judgements. Several previous works have used the embeddings from the penultimate layer of vision DNNs and showed that a reweighting of these features improves the fit between human similarity judgments and DNNs. These studies underline the idea that these embeddings form a good basis set but lack the correct level of salience. Here we re-examined the grounds for this idea and on the contrary, we hypothesized that these embeddings, beyond forming a good basis set, also have the correct level of salience to account for similarity judgments. It is just that the huge dimensional embedding needs to be pruned to select those features relevant for the considered domain for which a similarity space is modeled. In Study 1 we supervised DNN pruning based on a subset of human similarity judgments. We found that pruning: i) improved out-of-sample prediction of human similarity judgments from DNN embeddings, ii) produced better alignment with WordNet hierarchy, and iii) retained much higher classification accuracy than reweighting. Study 2 showed that pruning by neurobiological data is highly effective in improving out-of-sample prediction of brain-derived representational dissimilarity matrices from DNN embeddings, at times fleshing out isomorphisms not otherwise observable. Pruning supervised by human brain/behavior therefore effectively identifies alignable dimensions of semantic knowledge between DNNs and humans and constitutes an effective method for understanding the organization of knowledge in neural networks.

show abstract

Section: Meg Resultssupporting

confidence: 89%

Section: Methods: Learning Pruning From Meg Datamentioning

confidence: 99%

Improved prediction of behavioral and neural similarity spaces using pruned DNNs

Tarigopula

Fairhall

Hasson

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Future studies could employ tasks, such as category verification or story listening/reading (Huth et al, 2016;Deniz et al, 2019;Popham et al, 2021) that encourage deep processing of words and their context in modality-independent rather than modality-focused details. Third, we cannot exclude the possibility that M/EEG scalp sensor patterns lack the sensitivity to uncover the subtle signal differences essential for the readout of modality-unspecifc contents (Giari et al, 2020), while such differences can be revealed with spatially precise fMRI recording (Proklova et al, 2016(Proklova et al, , 2019.…”

Section: Discussionmentioning

confidence: 99%

Resolving the time course of visual and auditory object categorization

Iamshchinina

Karapetian

Kaiser

et al. 2021

Preprint

View full text Add to dashboard Cite

Humans can effortlessly categorize objects, both when they are conveyed through visual images and spoken words. To resolve the neural correlates of object categorization, studies have so far primarily focused on the visual modality. It is therefore still unclear how the brain extracts categorical information from auditory signals. In the current study we used EEG (N=47) and time-resolved multivariate pattern analysis to investigate (1) the time course with which object category information emerges in the auditory modality and (2) how the representational transition from individual object identification to category representation compares between the auditory modality and the visual modality. Our results show that (1) that auditory object category representations can be reliably extracted from EEG signals and (2) a similar representational transition occurs in the visual and auditory modalities, where an initial representation at the individual-object level is followed by a subsequent representation of the objects‘ category membership. Altogether, our results suggest an analogous hierarchy of information processing across sensory channels. However, we did not find evidence for a shared supra-modal code, suggesting that the contents of the different sensory hierarchies are ultimately modality-unique.

show abstract

“…Training was stopped at 25 iterations, yielding a final cost of 0.0060. To test the validity of our embeddings, two tests were performed: (1) we compared embeddings semantic relationships between words with human judgments for 160 basic objects from Giari et al (2020) and found a strong similarity [r=.39];…”

Section: Word Embeddingsmentioning

confidence: 99%

Convolutional neural networks reveal topic-level representations of sentences in medial prefrontal cortex, lateral anterior temporal lobe, precuneus, and angular gyrus

Acunzo¹,

Low²,

Fairhall³

2021

Preprint

View full text Add to dashboard Cite

When reading a sentence, individual words can be combined to create more complex meaning. In this study, we sought to uncover brain regions that reflect the representation of meaning at the sentence level, as opposed to only the meaning of their individual constituent words. Using fMRI, we recorded the neural activity of participants while reading sentences. We constructed sentence topic-level representations using the final layer of a convolutional neural network (CNN) trained to classify Wikipedia sentences into broad semantic categories. This model was contrasted with word-level sentence representations constructed using the average of the word embeddings constituting the sentence. Using representational similarity analysis, we found that the medial prefrontal cortex, lateral anterior temporal lobe, precuneus, and angular gyrus more strongly represent sentence topic-level, compared to word-level, meaning, uncovering the important role of these semantic system regions in the representation of integrated meaning. Conversely, these results validate the capacity of CNNs to capture human sentence-level representations.

show abstract

Spatiotemporal properties of the neural representation of conceptual content for words and pictures – an MEG study

Cited by 25 publications

References 33 publications

Improved prediction of behavioral and neural similarity spaces using pruned DNNs

Improved prediction of behavioral and neural similarity spaces using pruned DNNs

Resolving the time course of visual and auditory object categorization

Convolutional neural networks reveal topic-level representations of sentences in medial prefrontal cortex, lateral anterior temporal lobe, precuneus, and angular gyrus

Contact Info

Product

Resources

About