Commonality of neural representations of sentences across languages: Predicting brain activation during Portuguese sentence comprehension using an English-based model of brain function

Abstract

Keywords:Cross-language commonality, sentence representations in bilinguals, Predictive modeling of sentence representations, Meta-language brain locations in sentence processing ABSTRACTThe aim of the study was to test the cross-language generative capability of a model that predicts neural activation patterns evoked by sentence reading, based on a semantic characterization of the sentence. In a previous study on English monolingual speakers (Wang, Cherkassky, & Just, submitted), a computational model performed a mapping from a set of 42 concept-level semantic features (Neurally Plausible Semantic Features, NPSFs) as well as 6 thematic role markers to neural activation patterns (assessed with fMRI), to predict activation levels in a network of brain locations. The model used two types of information gained from the English-based fMRI data to predict the activation for individual sentences in Portuguese. First, it used the mapping weights from NPSFs to voxel activation levels derived from the model for English reading. Second, the brain locations for which the activation levels were predicted were derived from a factor analysis of the brain activation patterns during English reading. These meta-language locations were defined by the clusters of voxels with high loadings on each of the four main dimensions (factors), namely people, places, actions and feelings, underlying the neural representations of the stimulus sentences.This cross-language model succeeded in predicting the brain activation patterns associated with the reading of 60 individual Portuguese sentences that were entirely new to the model, attaining accuracies reliably above chance level. The prediction accuracy was not affected by whether the Portuguese speaker was monolingual or Portuguese-English bilingual. The model's confusion errors indicated an accurate capture of the events or states described in the sentence at a conceptual level. Overall, the cross-language predictive capability of the model demonstrates the neural commonality between speakers of different languages in the representations of everyday events and states, and provides an initial characterization of the common meta-language neural basis.

An Integrated Neural Decoder of Linguistic and Experiential Meaning

“…1). Although such additive composition is obviously an oversimplification that neglects the effects of word order, syntax, and morphology, it has endured as a practically successful technique in both computational linguistics (Mitchell and Lapata, 2010;Kiela and Clark, 2014), and fMRI analyses (Anderson et al, 2017aWang et al, 2017;Yang et al, 2017;Pereira et al, 2018). Indeed, attempts to incorporate other linguistic factors, such as syntax, into models have yet to make appreciable difference to neural decoding performance (Pereira et al, 2018;.…”

“…Other comparative analyses are in Anderson et al (2016) and Bulat et al (2017). fMRI data were decoded according to a commonly used leave-2-itemout cross-validation procedure (Mitchell et al, 2008;Chang et al, 2011;Sudre et al, 2012;Pereira et al, 2013Pereira et al, , 2018Wehbe et al, 2014;Anderson et al, 2016Anderson et al, , 2017aWang et al, 2017;Yang et al, 2017). At each cross-validation iteration, the 240 sentences were split into a test set of 2 sentences and a training set of 238 sentences.…”

“…The experiential model is based on peoples' ratings of their sensory/motor/affective/cognitive experiences with words and their referents (building on Cree and McRae, 2003;Vinson et al, 2003;Lynott and Connell, 2013). While experiential models have provided a basis for neural decoding (Chang et al, 2011;Fernandino et al, 2015Fernandino et al, , 2016Anderson et al, 2017aWang et al, 2017;Yang et al, 2017), it has never been clear whether and how textbased and experiential approaches complement one other. We newly demonstrate that text-based and experiential models differentially contribute to decoding sentences that do/do not contain linguistically oriented "abstract" words.…”

Stimulus-independent neural coding of event semantics: Evidence from cross-sentence fMRI decoding

“…However, in this study the classifier was not required to discriminate the content of the sentences, only whether they described a coherent event. In contrast, Yang et al (2017) developed a feature-based semantic model to code the content of 60 distinct English sentences. This model was trained with fMRI data from English speakers to form a predictive model of sentence activation patterns across the whole brain.…”

An essay on the language of thought

“…Another reason to assume a detachment between words and concepts, or between the phonological level and the semantic level, is suggested by some recent studies. Yang et al (2017), for instance, mapped the brain activity in English speakers when they read some sentences in their language and then fed the data to a computational model. They then recorded the brain activity from Portuguese speakers when they read sentences in Portuguese, and presented the model with their brain activity data.…”