fMRI reveals language-specific predictive coding during naturalistic sentence comprehension

Abstract

Much research in cognitive neuroscience supports prediction as a canonical computation of cognition in many domains. Is such predictive coding implemented by feedback from higher-order domain-general circuits, or is it locally implemented in domain-specific circuits? What information sources are used to generate these predictions? This study addresses these two questions in the context of language processing. We present fMRI evidence from a naturalistic comprehension paradigm (1) that predictive coding in the brain's response to language is domain-specific, and (2) that these predictions are sensitive both to local word co-occurrence patterns and to hierarchical structure. Using a recently developed deconvolutional time series regression technique that supports data-driven hemodynamic response function discovery from continuous BOLD signal fluctuations in response to naturalistic stimuli, we found we found effects of prediction measures in the language network but not in the domain-general, multiple-demand network. Moreover, within the language network, surface-level and structural prediction effects were separable. The predictability effects in the language network were substantial, with the model capturing over 37% of explainable variance on held-out data. These findings indicate that human sentence processing mechanisms generate predictions about upcoming words using cognitive processes that are sensitive to hierarchical structure and specialized for language processing, rather than via feedback from high-level executive control mechanisms. 4 2016; Lopopolo et al., 2017; see Table 1 for summary), a well-established predictor of behavioral measures in naturalistic language comprehension (These previous naturalistic studies of linguistic prediction effects in the brainusing estimates of prediction effort such as surprisal, the negative log probability of a word given its context, or entropy, an information-theoretic measure of the degree of constraint placed by the context on upcoming words (Hale, 2001)have yielded mixed results on the existence, type, and functional location of such effects. For example, of the lexicalized and unlexicalized (part-of-speech) bigram and trigram models of word surprisal explored in Brennan et al. (2016), only part-of-speech bigrams positively modulated neural responses in most regions of the functionally localized language network. Lexicalized bi-and trigrams and part-of-speech trigrams yielded generally null or negative results (16 out of 18 comparisons). By contrast, Willems et al. (2015) found lexicalized trigram effects in regions typically associated with language processing (e.g., anterior and posterior temporal lobe). In addition, Willems et al. (2015) and Lopopolo et al. (2017) found prediction effects in regions that are unlikely to be specialized for language processing, including (aggregating across both studies) the brain stem, amygdala, putamen, and hippocampus, as well as in superior frontal areas more typically associated with domain-general executive functions like...

Robust effects of working memory demand during naturalistic language comprehension in language-selective cortex

“…5-gram surprisal quantifies the predictability of words as the negative log probability of a word given the four words preceding it in context. PCFG and 5-gram surprisal were investigated by Shain, Blank, et al (2020) because their interpretable structure permitted testing of hypotheses of interest in that study. However, their strength as language models has now been outstripped by less interpretable but better performing incremental language models based on deep neural networks (Jozefowicz et al, 2016;Gulordava et al, 2018;Radford et al, 2019).…”

“…Following Shain, Blank, et al (2020), we use continuous-time deconvolutional regression (CDR; Shain & Schuler, 2019 to infer the shape of the hemodynamic response function (HRF) from data (Boynton et al, 1996;Handwerker et al, 2004). We assumed the following two-parameter HRF kernel based on the widely-used double-gamma canonical HRF (Lindquist et al where parameters α, β are fitted using black box variational Bayesian (BBVI) inference.…”

A hierarchy of linguistic predictions during natural language comprehension

“…3). Ear-550 lier model-based papers generally either used only 551 feature-unspecific word unexpectedness [12, 14, 34, 552 68, 69] or quantified predictions at a single level such 553as syntax[13, 24, 50,70], phonemes[15,29,30] or 554 semantics [26]. In all cases, predictions at a specific 555 level were based on a separate model that had to be 556 independently trained and often only incorporated 557 linguistic context in a limited way.…”

Le Petit Prince: A multilingual fMRI corpus using ecological stimuli

“…This limited our understanding of the neural processes of language comprehension to small typological domains. To complement monolingual datasets such as the Narrative Brain Dataset (NBD) 14 , the Alice Dataset 15 and the Mother of Unification Studies 16 , we collected a multilingual fMRI dataset consisted of Antoine de Saint-Exupéry's The Little Prince in English, Chinese and French. A total of 112 subjects (49 English speakers, 35 Chinese speakers and 28 French speakers) listened to the whole audiobook for about 100 minutes in the scanner (see Table 1 and Table 2 for the demographics of the participants, data collection procedures, and stimuli information for the English, Chinese, and French datasets).…”