Commonality of neural representations of words and pictures

Shinkareva, Svetlana V.; Malave, Vincente L.; Mason, Robert A.; Mitchell, Tom M.; Just, Marcel Adam

doi:10.1016/j.neuroimage.2010.10.042

Cited by 128 publications

(145 citation statements)

References 47 publications

Supporting

Mentioning

132

Contrasting

Order By: Relevance

“…We also observed significantly reduced activity in left angular gyrus, left middle frontal gyrus, and bilateral orbital frontal cortex, areas that have been suggested to be involved in visual semantics (Gerlach, 2007) but that are outside of what is considered the core visual category-selective perception network. Interestingly these areas have been strongly implicated in the semantics of language (Binder, Desai, Graves, & Conant, 2009;Price, 2012), and there is evidence for a commonality of neural representations of words and pictures (Bright, Moss, & Tyler, 2004;Chee et al, 2000;Gates & Yoon, 2005;Shinkareva, Malave, Mason, Mitchell, & Just, 2011).…”

Section: Category Regularitiesmentioning

confidence: 99%

Predictability of what or where reduces brain activity, but a bottleneck occurs when both are predictable

Davis

Hasson

2018

NeuroImage

View full text Add to dashboard Cite

Detecting regularities in the sensory environment licenses predictions that enable adaptive behaviour. However, it is unclear whether predictions about object category, location, or both dimensions are mediated by overlapping systems, and relatedly, whether constructing predictions about both category and location is associated with processing bottlenecks. To examine this issue, in an fMRI study, we presented participants with image-series in which non-deterministic transition probabilities enabled predictions about either the location of the next image, its semantic category, both dimensions, or neither (the latter forming a "no-regularity" random baseline condition). Speaking to a common system, all three predictable conditions resulted in reduced BOLD activity in four clusters: left rostral anterior cingulate cortex; bilateral putamen, caudate and thalamus; right precentral gyrus, and left visual cortex. Pointing to a processing bottleneck, in some regions, a significant interaction between the two factors was found whereby category-predictable series were associated with lower activity -but only when location regularity was absent. Finally, category regularity decreased activation in areas of the ventral visual stream and semantic areas of lateral temporal cortex, and location regularity decreased activation in a dorsal fronto-parietal cluster, long implicated in the endogenous control of spatial attention. Our findings confirm and expand a role for dACC/dmPFC and striatum in monitoring or responding to uncertainty in the environment and point to a limited capacity bottleneck when multiple predictions are concurrently licensed.

show abstract

Section: Category Regularitiesmentioning

confidence: 99%

Predictability of what or where reduces brain activity, but a bottleneck occurs when both are predictable

Davis

Hasson

2018

NeuroImage

View full text Add to dashboard Cite

show abstract

“…test set). Previous studies of semantic representations used this method to decode the semantic category of words from brain activations patterns, and generalize this categorical discrimination across different input formats (from pictures to words and vice versa) (Shinkareva et al, 2011;Simanova et al, 2014).…”

Section: Multivariate Pattern Analysesmentioning

confidence: 99%

“…Researchers capitalizing from both machine learning techniques and Representational Similarity Analysis (RSA) frameworks have shown that it is possible to discriminate between words belonging to different semantic categories (e.g., animals vs tools) as well as sub-categorical clusters (e.g., mammals vs insects) using distributed patterns of brain activation (Shinkareva et al, 2011;Bruffaerts et al, 2013;Devereux et al, 2013;Fairhall and Caramazza, 2013;Simanova et al, 2014), but they did not determine if such discriminations were driven by conceptual or/and by correlated perceptual information (Naselaris and Kay, 2015). Finally, the so called "encoding" approach (modelling and predicting voxel-wise activation for different stimuli according to their defining set of features) has been successfully applied to predict brain activation during the elaboration of images and movies (Naselaris et al, 2009;Nishimoto et al, 2011), and only very recently to words (Fernandino et al, 2015a).…”

mentioning

confidence: 99%

Word meaning in the ventral visual path: a perceptual to conceptual gradient of semantic coding

et al. 2016

View full text Add to dashboard Cite

valentinaborghesani@gmail.comThe authors declare no competing financial interests. ABSTRACTThe meaning of words referring to concrete items is thought of as a multidimensional representation that includes both perceptual (e.g., average size, prototypical color) and conceptual (e.g., taxonomic class) dimensions. Are these different dimensions coded in different brain regions? In healthy human subjects, we tested the presence of a mapping between the implied real object size (a perceptual dimension) and the taxonomic categories at different levels of specificity (conceptual dimensions) of a series of words, and the patterns of brain activity recorded with functional magnetic resonance imaging in six areas along the ventral occipito-temporal cortical path. Combining multivariate pattern classification and representational similarity analysis, we found that the real object size implied by a word appears to be primarily encoded in early visual regions, while the taxonomic category and subcategorical cluster in more anterior temporal regions. This anteroposterior gradient of information content indicates that different areas along the ventral stream encode complementary dimensions of the semantic space.

show abstract

“…An ever increasing number of studies have succeeded in detecting patterns of brain activity that reflect the identity or the semantic category of pictures (Haxby et al, 2001;Hanson et al, 2004;Shinkareva et al, 2008;Connolly et al, 2012;Mur et al, 2012;Bruffaerts et al, 2013a), written words Just et al, 2010), written words and pictures (Shinkareva et al, 2011;Simanova et al, 2012;Bruffaerts et al, 2013b;Devereux et al, 2013;Fairhall and Caramazza, 2013), written and spoken words (Akama et al, 2012;Simanova et al, 2012) as well as natural sounds (Simanova et al, 2012).…”

Section: Introductionmentioning

confidence: 99%

Left perirhinal cortex codes for similarity in meaning between written words: Comparison with auditory word input

et al. 2015

View full text Add to dashboard Cite

This is a pre-copy-editing, author-produced PDF of an article accepted for publication in Neuropsychologia following peer review. The definitive publisher-authenticated version (Liuzzi A.G., Bruffaerts R., Dupont P., Adamczuk K., Peeters R., De Deyne S., Storms G., Vandenberghe R. AbstractLeft perirhinal cortex has been previously implicated in associative coding. According to a recent experiment, the similarity of perirhinal fMRI response patterns to written concrete words is higher for words which are more similar in their meaning. If left perirhinal cortex functions as an amodal semantic hub, one would predict that this semantic similarity effect would extend to the spoken modality. We conducted an event-related fMRI experiment and evaluated whether a same semantic similarity effect could be obtained for spoken as for written words. Twenty healthy subjects performed a property verification task in either the written or the spoken modality. Words corresponded to concrete animate entities for which extensive feature generation was available from more than 1000 subjects. From these feature generation data, a concept-feature matrix was derived which formed the basis of a cosine similarity matrix between the entities reflecting their similarity in meaning (called the "semantic cossimilarity matrix"). Independently, we calculated a cosine similarity matrix between the left perirhinal fMRI activity patterns evoked by the words (called the "fMRI cossimilarity matrix"). Next, the similarity was determined between the semantic cossimilarity matrix and the fMRI cossimilarity matrix. This was done for written and spoken words pooled, for written words only, for spoken words only, as well as for crossmodal pairs. Only for written words did the fMRI cossimilarity matrix correlate with the semantic cossimilarity matrix. Contrary to our prediction, we did not find any such effect for auditory word input nor did we find cross-modal effects in perirhinal cortex between written and auditory words. Our findings situate the contribution of left perirhinal cortex to word processing at the top of the visual processing pathway, rather than at an amodal stage where visual and auditory word processing pathways have already converged.

show abstract

Commonality of neural representations of words and pictures

Cited by 128 publications

References 47 publications

Predictability of what or where reduces brain activity, but a bottleneck occurs when both are predictable

Predictability of what or where reduces brain activity, but a bottleneck occurs when both are predictable

Word meaning in the ventral visual path: a perceptual to conceptual gradient of semantic coding

Left perirhinal cortex codes for similarity in meaning between written words: Comparison with auditory word input

Contact Info

Product

Resources

About