Robust effects of working memory demand during naturalistic language comprehension in language-selective cortex

Abstract

A standard view of human language processing is that comprehenders build richly structured mental representations of natural language utterances, word by word, using computationally costly memory operations supported by domain-general working memory resources. However, three core claims of this view have been questioned, with some prior work arguing that (1) rich word-by-word structure building is not a core function of the language comprehension system, (2) apparent working memory costs are underlyingly driven by word predictability (surprisal), and/or (3) language comprehension relies primarily on domain-general rather than domain-specific working memory resources. In this work, we simultaneously evaluate all three of these claims using naturalistic comprehension in fMRI. In each participant, we functionally localize (a) a language-selective network and (b) a 'multiple-demand' network that supports working memory across domains, and we analyze the responses in these two networks of interest during naturalistic story listening with respect to a range of theory-driven predictors of working memory demand under rigorous surprisal controls. Results show robust surprisal-independent effects of word-by-word memory demand in the language network and no effect of working memory demand in the multiple demand network. Our findings thus support the view that language comprehension (1) entails word-by-word structure building using (2) computationally intensive memory operations that are not explained by surprisal. However, these results challenge (3) the domain-generality of the resources that support these operations, instead indicating that working memory operations for language comprehension are carried out by the same neural resources that store linguistic knowledge.

Graded sensitivity to structure and meaning throughout the human language network

“…In summary, our finding of lexicality effects (and/or constituent length by lexicality interactions) in inferior frontal and posterior temporal language regions undermines an influential claim in PDD: that these regions support abstract, content-independent syntactic structure building. Our results are instead consistent with growing evidence that linguistic representations and computations over a range of levels of description (phonological, lexical, syntactic, and semantic) are highly distributed across the language network and are not spatially segregated (10,52,54,74,78,79).…”

“…To test whether PDD's results are approximated by independently motivated word-by-word measures of processing demand, we considered six alternative measures from the psycholinguistic literature: four of them are derived from memory-based accounts of sentence processing (75,79,92), and the other two-from surprisal-based accounts (93)(94)(95). The mean value of each predictor by condition is plotted in Figure 1C.…”

Similarity of Computations Across Domains Does Not Imply Shared Implementation: The Case of Language Comprehension

“…(Darker gray bars show results from passive reading and listening experiments, and lighter bars show results from experiments in which language processing was accompanied by a secondary task (e.g., a memory probe, comprehension, or sentence judgment task). The graph in (e) shows effects of 5-gram surprisal (Shain et al, 2020), syntactic parser-based surprisal (i.e., surprisal derived from a computational model of syntactic structure building; Shain et al, 2020), and integration cost (Shain et al, 2021) in each network during naturalistic story comprehension. Integration cost was operationalized as in the dependency locality theory (DLT; Gibson, 2000).…”

“…Given that working memory is thought to be one of the core functions supported by the MD network (Duncan et al, 2020), one plausible role for this network in language comprehension is as a working memory resource for syntactic structure building. We (Shain et al, 2021) investigated this possibility by exploring the contribution of multiple theory-derived measures of working memory cost to explaining variance in the language and MD networks' responses to naturalistic linguistic stimuli. The language network showed a systematic and generalizable (to an unseen data portion) response to variants of integration cost as proposed by Gibson's (2000) dependency locality theory.…”

LanA (Language Atlas): A probabilistic atlas for the language network based on fMRI data from >800 individuals

“…The correlation values were calculated within the language 'parcels'-masks that denote typical locations of language areas. These masks (available at http://evlab.mit.edu/funcloc) were derived from a probabilistic language atlas based on 220 participants (a subset of the participants in the current set of 806) and have been used in much past work (e.g., Diachek, Blank, Siegelman et al, 2020;Ivanova et al, 2020;Jouravlev et al, 2019Jouravlev et al, , 2020Mollica et al, 2020;Shain, Blank et al, 2021;Wehbe et al, 2021). Six masks (three in the frontal cortex and three in the temporal and parietal cortex) were derived from the probabilistic atlas in the left hemisphere and mirror-projected onto the right hemisphere.…”