Robust Effects of Working Memory Demand during Naturalistic Language Comprehension in Language-Selective Cortex

Abstract

A standard view of human language processing is that comprehenders build richly structured mental representations of natural language utterances, word by word, using computationally costly memory operations supported by domain-general working memory resources. However, three core claims of this view have been questioned, with some prior work arguing that (1) rich word-by-word structure building is not a core function of the language comprehension system, (2) apparent working memory costs are underlyingly driven by word predictability (surprisal), and/or (3) language comprehension relies primarily on domain-general rather than domain-specific working memory resources. In this work, we simultaneously evaluate all three of these claims using naturalistic comprehension in fMRI. In each participant, we functionally localize (a) a language-selective network and (b) a 'multiple-demand' network that supports working memory across domains, and we analyze the responses in these two networks of interest during naturalistic story listening with respect to a range of theory-driven predictors of working memory demand under rigorous surprisal controls. Results show robust surprisal-independent effects of word-by-word memory demand in the language network and no effect of working memory demand in the multiple demand network. Our findings thus support the view that language comprehension (1) entails word-by-word structure building using (2) computationally intensive memory operations that are not explained by surprisal. However, these results challenge (3) the domain-generality of the resources that support these operations, instead indicating that working memory operations for language comprehension are carried out by the same neural resources that store linguistic knowledge.

Precision fMRI reveals that the language-selective network supports both phrase-structure building and lexical access during language production

“…The data pattern we observe here, along with the findings reported in past studies of language comprehension (e.g., Fedorenko et al, 2010, 2012a, 2020; Pallier et al, 2011; Blank et al, 2016; Shain, Blank et al, 2020)—whereby phrase-structure building demands elicit strong responses across the language network during both comprehension and production—aligns with recent evidence from Shain et al (in press). Shain and colleagues report a large-scale fMRI study that suggests that language comprehension involves computationally demanding word-by-word structure building operations even when participants passively listen to naturalistic stories.…”

“…The response to phrase-structure building demands (evidenced by a stronger response during sentence production than word-list production) was reliable in every language fROI. This distributed nature of phrase-structure building a) parallels the distributed effects of syntactic demands during language comprehension (e.g., Blank et al, 2016; Shain, Blank et al, 2020; Shain et al, in press), and b) aligns with evidence from aphasia, where damage to both frontal and temporal language areas and the white matter tracts connecting them can result in syntactic deficits (e.g., Kempler et al, 1991; Caplan et al, 1996; Dick et al, 2001; Wilson & Saygin, 2004; Mesulam et al, 2015; Wilson et al, 2022; see deBleser, 1987 for a discussion of earlier evidence), thus adding to the growing evidence against focal implementation of combinatorial linguistic processing. Importantly, we showed that the sentence > word-list effect in production cannot be explained by general cognitive demands.…”

“…Swets et al, 2013; Goldberg & Ferreira, 2022), comprehension is possible even when word order and other morpho-syntactic cues are degraded or absent (e.g., Ferreira et al, 2002; Ferreira, 2003; Levy, 2008; Levy et al, 2009; Gibson et al, 2013; Ferreira & Lowder, 2016; Mollica et al, 2020; Mahowald, Diachek et al, 2022; cf. Shain et al, in press for evidence that comprehenders compute fine-grained syntactic representations even during passive listening in naturalistic paradigms). Matchin & Hickok (2019), based largely on data from aphasia, recently proposed that such a mechanism may be housed in the inferior frontal component of the language network (in contrast with its posterior temporal component, which they hypothesized supports morpho-syntactic demands in both comprehension and production).…”

Neural populations in the language network differ in the size of their temporal receptive windows

“…In spite of some claims about regional dissociation between lexico-semantic and syntactic/combinatorial processing based on this and other paradigms(e.g., Dapretto and Bookheimer, 1999; Embick et al, 2000; Friederici et al, 2000; Kuperberg et al, 2000; see Fedorenko et al, 2020 for a recent review of this literature), studies that rely on robust individual-subject analyses have shown that these two aspects of language processing do not dissociate: all areas of the language network show a profile whereby the response is strongest to sentences, lower to lists of words and Jabberwocky sentences, and lowest to lists of nonwords (e.g., Fedorenko et al, 2010; Bedny et al, 2011; Shain et al, 2021; see Dick et al, 2001 for early arguments against the dissociation between lexical and syntactic processing). This ubiquitous sensitivity to both word meanings and syntactic structure building aligns with studies that have reported robust sensitivity to structure building across all parts of the language network (Blank et al, 2016; Fedorenko et al, 2020; Shain, Blank et al, 2020; Shain et al 2022a) and supports views of language whereby sentence structure building is deeply intertwined with the processing of word meanings (e.g., Bybee, 1999, 2013; Goldberg, 2003; Jackendoff, 2007; Arnon and Snider, 2010; Jackendoff and Audring, 2020).…”

“Um…, it’s really difficult to… um… speak fluently”: Neural tracking of spontaneous speech

“…Several metrics have been proposed in the past for quantifying syntactic complexity on a continuous basis (Gibson, 1998; Szmrecsanyi, 2004; Roark et al, 2007, 2011; Baumann, 2014; Jing and Liu, 2015; Ambati et al, 2016). In attempt to move towards studying language under more naturalistic conditions, such metrics can be useful when studying uncontrolled stimuli such as speech, as recently been shown by studies that associated such scores with fluctuations in the neural response (Brennan et al, 2016; Nelson et al, 2017; Lopopolo et al, 2021; Shain et al, 2021). Here we used the mean-dependency distance (MDD), computed per clause, which has been hypothesized to reflect working memory demands (Liu, 2008; Gildea and Temperley, 2010; Collins, 2014; Futrell et al, 2015; Liu et al, 2017), yet to our knowledge, the neural correlates of MDD have not been tested before.…”