Modeling fMRI time courses with linguistic structure at various grain sizes

Abstract

Neuroimaging while participants listen to audiobooks provides a rich data source for theories of incremental parsing. We compare nested regression models of these data. These mixed-effects models incorporate linguistic predictors at various grain sizes ranging from part-of-speech bigrams, through surprisal on context-free treebank grammars, to incremental node counts in trees that are derived by Minimalist Grammars. The fine-grained structures make an independent contribution over and above coarser predictors. However, this result only obtains with time courses from anterior temporal lobe (aTL). In analogous time courses from inferior frontal gyrus, only n-grams improve upon a non-syntactic baseline. These results support the idea that aTL does combinatoric processing during naturalistic story comprehension, processing that bears a systematic relationship to linguistic structure.

Language structure in the brain: A fixation-related fMRI study of syntactic surprisal in reading

“…More consistent with the present study's focus on syntactic difficulty, Hale et al (2015) focused on linguistic prediction. They presented Alice in Wonderland via speech and examined linguistic predictors across representational grain size.…”

“…Two recently published fMRI studies of surprisal in language processing during story listening reported related results (Hale et al, 2015;Willems et al, in press; see also Bachrach, 2008, for a related project). Although the general strategy of using surprisal and fMRI in these studies was similar to that taken in the present study, they differed in a number of critical ways, including the input modality (speech versus written text) and the type of surprisal investigated.…”

“…For example, Hale et al (2015) and Brennan et al (2010) operationalized syntactic complexity as a count of the number of syntactic nodes needed to integrate each word into the phrase structure, relating difficulty to the density of syntactic nodes associated with a word (Frazier, 1985;Hawkins, 1994). Another approach might be to implement a metric based on the dependency-locality theory, which relates processing difficulty to the amount of integration required and the distance over which that integration must take place (Gibson, 1998(Gibson, , 2000.…”

“…Higher surprisal values are associated with greater processing difficulty: Surprisal values reliably predict word-by-word reading times derived from eye movement measures (e.g., Boston et al, 2008;Demberg and Keller, 2008;Demberg et al, 2013;Levy, 2008), the N400 event-related potential (ERP) component (Frank et al, 2015), and fMRI activation (Hale et al, 2015;Willems et al, in press).…”

Word predictability and semantic similarity show distinct patterns of brain activity during language comprehension

“…Indeed, it has repeatedly been shown that surprisal predicts word-reading time (Frank & Thompson, 2012;Monsalve, Frank, & Vigliocco, 2012;Smith & Levy, 2013). Surprisal effects have also been found in brain imaging data: Higher surprisal value results in a stronger N400 ERP component (Frank, Otten, Galli, & Vigliocco, 2015) as well as its magnetoencephalography (MEG) equivalent (Parviz, Johnson, Johnson, & Brock, 2011;Wehbe, Vaswani, Knight, & Mitchell, 2014), and stronger Blood-Oxygenation Level Dependent (BOLD) response in anterior temporal cortex, inferior frontal gyrus, and the visual word-form area (VWFA; Hale, Lutz, Luh, & Brennan, 2015;Willems, Frank, Nijhof, Hagoort & Van den Bosch, 2016). …”