Tracking neural coding of perceptual and semantic features of concrete nouns

Abstract

We present a methodological approach employing magnetoencephalography (MEG) and machine learning techniques to investigate the flow of perceptual and semantic information decodable from neural activity in the half second during which the brain comprehends the meaning of a concrete noun. Important information about the cortical location of neural activity related to the representation of nouns in the human brain has been revealed by past studies using fMRI. However, the temporal sequence of processing from sensory input to concept comprehension remains unclear, in part because of the poor time resolution provided by fMRI. In this study, subjects answered 20 questions (e.g. is it alive?) about the properties of 60 different nouns prompted by simultaneous presentation of a pictured item and its written name. Our results show that the neural activity observed with MEG encodes a variety of perceptual and semantic features of stimuli at different times relative to stimulus onset, and in different cortical locations. By decoding these features, our MEG-based classifier was able to reliably distinguish between two different concrete nouns that it had never seen before. The results demonstrate that there are clear differences between the time course of the magnitude of MEG activity and that of decodable semantic information. Perceptual features were decoded from MEG activity earlier in time than semantic features, and features related to animacy, size, and manipulability were decoded consistently across subjects. We also observed that regions commonly associated with semantic processing in the fMRI literature may not show high decoding results in MEG. We believe that this type of approach and the accompanying machine learning methods can form the basis for further modeling of the flow of neural information during language processing and a variety of other cognitive processes.

BrainBench: A Brain-Image Test Suite for Distributional Semantic Models

“…Our process of computing 2 vs. 2 accuracy over rows of a correlation matrix is different than the original methodology for these datasets (Mitchell et al, 2008;Sudre et al, 2012). Previous work trained regression models that took brain images as input and predicted the dimensions of a DS model as output.…”

“…For BrainBench we use two brain image datasets collected while participants viewed 60 concrete nouns with line drawings (Mitchell et al, 2008;Sudre et al, 2012). One dataset was collected using fMRI (Functional Magnetic Resonance Imaging) and one with MEG (Magnetoencephalography).…”

“…As there is a possibility of confounding visual properties with semantic properties, we have attempted to remove the activity attributable to visual properties from the brain images. In total we have 11 visual features which include things like the length of the word, the number of white pixels, and features of the line drawing (Sudre et al, 2012). To remove the visual stimulus' contribution to the signal, we train a regression model that predicts the signal in each brain image feature as a function of the 11 visual features.…”

Exploratory analysis of semantic categories: comparing data-driven and human similarity judgments

“…Another approach is to use a taxonomy or ontology as a basis for the similarity calculations (Seco et al 2004). A new prominent evaluation direction is comparing corpusderived vector representations to brain imaging results obtained with functional Magnetic Resonance Imaging (fMRI) (Mitchell et al 2008;Murphy et al 2012) or magnetoencephalography (MEG) (Sudre et al 2012). Direct vector space model evaluation concentrates on VSM performance, and measures the similarities of given words in the VSM model, and require human-annotated sources.…”

Stages of Processing in Associative Recognition: Evidence from Behavior, EEG, and Classification

“…To classify the data, we followed the methodology outlined by Sudre et al (2012), who used a classifier in combination with MEG data to investigate when perceptual and semantic features of nouns were reflected in neural activity. We first discuss how we preprocessed the data for the classifier, and then we describe the algorithm and the training/testing methodology.…”