Decoding semantic representations in mind and brain

Frisby, Saskia L; Halai, Ajay D.; Cox, Christopher R.; Ralph, Matthew A. Lambon; Rogers, Timothy T.

doi:10.1016/j.tics.2022.12.006

Cited by 24 publications

(20 citation statements)

References 141 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…What then do distributional semantic models provide for us in the context of our study? Although they do not capture the conceptual richness of the words as they functioned when originally uttered, nor for that matter nonlinguistic, sensorimotor experiences that undoubtedly shape meaning (Binder et al., 2016; Frisby, Halai, Cox, Lambon Ralph, & Rogers, 2022), they do allow insight into the semantic relatedness between utterances that are not restricted to strict lexical overlap. But it is important to keep in mind that this semantic relatedness is ultimately that which exists in a high‐dimensional vector space derived from pre‐existing text (in the case of our use of word2vec, billions of words from Google News).…”

Section: Discussionmentioning

confidence: 99%

Multi‐Level Linguistic Alignment in a Dynamic Collaborative Problem‐Solving Task

Duran,

Paige,

D'Mello

2024

Cognitive Science

View full text Add to dashboard Cite

Cocreating meaning in collaboration is challenging. Success is often determined by people's abilities to coordinate their language to converge upon shared mental representations. Here we explore one set of low‐level linguistic behaviors, linguistic alignment, that both emerges from, and facilitates, outcomes of high‐level convergence. Linguistic alignment captures the ways people reuse, that is, “align to,” the lexical, syntactic, and semantic forms of others' utterances. Our focus is on the temporal change of multi‐level linguistic alignment, as well as how alignment is related to communicative outcomes within a unique collaborative problem‐solving paradigm. The primary task, situated within a virtual educational video game, requires creative thinking between three people where the paths for possible solutions are highly variable. We find that over time interactions are marked by decreasing lexical and syntactic alignment, with a trade‐off of increasing semantic alignment. However, greater semantic alignment does not translate into better team performance. Overall, these findings provide greater clarity on the role of linguistic coordination within complex and dynamic collaborative problem‐solving tasks.

show abstract

Section: Discussionmentioning

confidence: 99%

Multi‐Level Linguistic Alignment in a Dynamic Collaborative Problem‐Solving Task

Duran,

Paige,

D'Mello

2024

Cognitive Science

View full text Add to dashboard Cite

show abstract

“…In this sense, the selection of a regularization function amounts to a prior hypothesis stipulating how the neural signal is likely to be structured. As recently argued in (Frisby et al, 2023), the choice of regularizer should be informed by explicit hypotheses about the nature of the signal to be decoded-in this case, a hypothesis about how neurophysiological signals measured by surface electrodes in the brain might encode a graded, multi-variate semantic space.…”

Section: Discussionmentioning

confidence: 99%

“…Yet direct empirical tests of this proposal—representational similarity analysis (RSA) of functional imaging data collected while people perform semantic tasks on words or pictures—have not generally tended to support it. A recent review identifies 24 studies applying RSA to uncover semantic representations in the brain (Frisby et al, 2023); of these, 18 (75%) failed to identify semantic structure in the anterior temporal cortex (for the exceptions, see Bruffaerts et al, 2013; Devereux et al, 2018; Fairhall & Caramazza, 2013; Martin et al, 2018; Peelen & Caramazza, 2012). Many of these studies instead find that semantic structure is encoded in brain areas not otherwise thought to be critical to semantic representation, including posterior cortical regions (Connolly et al, 2012), inferior and superior frontal and motor cortex (Carota et al, 2017; Wang et al, 2017), the left pars triangularis (Liuzzi et al, 2017), right superior parietal cortex (Wang et al, 2017), the insula and occipeto-parietal cortex (Kivisaari et al, 2019), and the posterior cingulate cortex (Fairhall & Caramazza, 2013).…”

Section: Introductionmentioning

confidence: 99%

Representational similarity learning reveals a graded multidimensional semantic space in the human anterior temporal cortex

Cox

Rogers

Shimotake

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Neuro-cognitive models of semantic memory have proposed that the ventral anterior temporal lobes (vATLs) encode a graded, distributed, and multidimensional semantic space--yet neuroimaging studies seeking brain regions that encode semantic structure rarely identify these areas. In computer simulations we show that this discrepancy may arise from limitations of a commonly-used multivariate analysis, and introduce a new approach, representational similarity learning (RSL), that resolves these. We then use RSL to decode semantic structure from ECoG data collected from the vATL cortical surface while participants named line drawings of common items. The results reveal a graded, multi-dimensional semantic space encoded in neural activity across the vATL, which evolves over time and simultaneously expresses both broad and finer-grained semantic structure amongst animate and inanimate concepts. The work thus resolves the apparent discrepancy within the semantic cognition literature and suggests a new approach to discovering representational structure in neural data more generally.

show abstract

“…The timing of the transition from visual representations of individual objects to more abstract semantic-type concepts is determined by identifying the time points at which the category of the object can be accurately predicted from the multivariate representation in the MEG and EEG data (Proklova et al, 2016;Kumar et al, 2017). However, drawing meaningful conclusions from the above-chance classification using stimuli from a single modality may be limited, as there are often multiple dimensions in which the two conditions differ (Peelen and Downing, 2022;Frisby et al, 2023). For example, one can argue that perceptual information beyond semantic content influences classification outcomes.…”

Section: Introductionmentioning

confidence: 99%

Spatiotemporal properties of common semantic categories for words and pictures

Bezsudnova,

Quinn,

Jensen

2023

Preprint

View full text Add to dashboard Cite

The timing of semantic processing during object recognition in the brain is a topic of ongoing discussion. One way of addressing this question is by applying multivariate pattern analysis (MVPA) to human electrophysiological responses to object images of different semantic categories. However, while MVPA can reveal whether neuronal activity patterns are distinct for different stimulus categories, concerns remain on whether low-level visual features also contribute to the classification results. To circumvent this issue, we applied a cross-decoding approach to magnetoencephalography (MEG) data from stimuli from two different modalities: images and their corresponding written words. We employed items for three categories and presented them in a randomized order. We show that If the classifier is trained on words, pictures are classified between 150 - 430 ms from after stimulus onset, and when training on pictures, words are classified between 225 - 430 ms. The topographical map identified using a searchlight approach for cross-modal activation in both directions showed left lateralization confirming the involvement of linguistics representations. These results point to semantic activation of pictorial stimuli occurring at ≈150 ms whereas for words the semantic activation occurs at ≈230 ms.

show abstract

Decoding semantic representations in mind and brain

Cited by 24 publications

References 141 publications

Multi‐Level Linguistic Alignment in a Dynamic Collaborative Problem‐Solving Task

Multi‐Level Linguistic Alignment in a Dynamic Collaborative Problem‐Solving Task

Representational similarity learning reveals a graded multidimensional semantic space in the human anterior temporal cortex

Spatiotemporal properties of common semantic categories for words and pictures

Contact Info

Product

Resources

About