Neural decoding of semantic concepts: a systematic literature review

Abstract

Objective Semantic concepts are coherent entities within our minds. They underpin our thought processes and are a part of the basis for our understanding of the world. Modern neuroscience research is increasingly exploring how individual semantic concepts are encoded within our brains and a number of studies are beginning to reveal key patterns of neural activity that underpin specific concepts. Building upon this basic understanding of the process of semantic neural encoding, neural engineers are beginning to explore tools and methods for semantic decoding: identifying which semantic concepts an individual is focused on at a given moment in time from recordings of their neural activity. In this paper we review the current literature on semantic neural decoding. Approach We conducted this review according to the Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) guidelines. Specifically, we assess the eligibility of published peer-reviewed reports via a search of PubMed and Google Scholar. We identify a total of 74 studies in which semantic neural decoding is used to attempt to identify individual semantic concepts from neural activity. Results Our review reveals how modern neuroscientific tools have been developed to allow decoding of individual concepts from a range of neuroimaging modalities. We discuss specific neuroimaging methods, experimental designs, and machine learning pipelines that are employed to aid the decoding of semantic concepts. We quantify the efficacy of semantic decoders by measuring information transfer rates. We also discuss current challenges presented by this research area and present some possible solutions. Finally, we discuss some possible emerging and speculative future directions for this research area. Significance Semantic decoding is a rapidly growing area of research. However, despite its increasingly widespread popularity and use in neuroscientific research this is the first literature review focusing on this topic across neuroimaging modalities and with a focus on quantifying the efficacy of semantic decoders.

Bimodal electroencephalography-functional magnetic resonance imaging dataset for inner-speech recognition

“…In both binary and multiclass (8) unimodal classification, performance was consistently lower for fMRI data than for EEG. This is initially surprising as Ryba et al [13] observed that fMRI decoders show better performance on semantic categories with higher conceptual similarity, such as 'child', 'wife', 'father', than neuroimaging techniques with lower spatial resolution like EEG. We suggest that in this current study superior multiclass performance with EEG is facilitated by the temporal features of inner-speech, and EEG's high temporal resolution.…”

“…The simultaneous data from 19 subjects (age: 26.63±2. 13) was collected for overt and inner speech. The EEG and fNIRS signals were preprocessed and used to train CNN-GRU based deep learning architecture to predict the overt and inner speech.…”

Decoding Semantics from Dynamic Brain Activation Patterns: From Trials to Task in EEG/MEG Source Space

“…e N e u r o A c c e p t e d M a n u s c r i p t previous computation stages (e.g., visual processing). While the majority of neural semantic decoders still focus on fMRI data (Rybář & Daly, 2022), MEG data contains rich information that can be exploited with multivariate analyses both from a temporal point of view and, when performed in source space, from a spatial point of view (e.g., Kietzmann et al (2019).…”

Modeling Brain Representations of Words' Concreteness in Context Using GPT‐2 and Human Ratings

“…Brain encoding studies, where machine learning is used to predict patterns of brain activity by learning functions from computational representations-for example Abraham et al (2014), Grootswagers, Wardle, & Carlson (2017), Haxby et al (2001), Haxby, Connolly, Guntupalli, & others (2014), Haynes (2015), Kragel, Koban, Barrett, & Wager (2018), Lemm, Blankertz, Dickhaus, & Müller (2011), Naselaris, Kay, Nishimoto, & Gallant (2011), Pereira, Mitchell, & Botvinick (2009), Rybář & Daly (2022)-have recently started to make use of vectorial models of meaning proposed in NLP that have been shown to capture an extremely wide range of information involved with semantic processing (for comprehensive reviews, see Hale et al, 2022;Murphy, Wehbe, & Fyshe, 2018). In encoding, vectorial semantic representations open new possibilities for the investigation of semantic processing in the brain (Bruffaerts et al, 2019;Diedrichsen & Kriegeskorte, 2017;Kay, 2018;Kriegeskorte, Mur, & Bandettini, 2008;Naselaris & Kay, 2015).…”