Dimensionality and ramping: Signatures of sentence integration in the dynamics of brains and deep language models

Abstract

A sentence is more than the sum of its words: its meaning depends on how they combine with one another. The brain mechanisms underlying such semantic composition remain poorly understood. To shed light on the neural vector code underlying semantic composition, we introduce two hypotheses: First, the intrinsic dimensionality of the space of neural representations should increase as a sentence unfolds, paralleling the growing complexity of its semantic representation, and second, this progressive integration should be reflected in ramping and sentence-final signals. To test these predictions, we designed a dataset of closely matched normal and Jabberwocky sentences (composed of meaningless pseudo words) and displayed them to deep language models and to 11 human participants (5 men and 6 women) monitored with simultaneous magneto-encephalography and intracranial electro-encephalography. In both deep language models and electrophysiological data, we found that representational dimensionality was higher for meaningful sentences than Jabberwocky. Furthermore, multivariate decoding of normal versus Jabberwocky confirmed three dynamic patterns: (i) a phasic pattern following each word, peaking in temporal and parietal areas, (ii) a ramping pattern, characteristic of bilateral inferior and middle frontal gyri, and (iii) a sentence-final pattern in left superior frontal gyrus and right orbitofrontal cortex. These results provide a first glimpse into the neural geometry of semantic integration and constrain the search for a neural code of linguistic composition.

Driving and suppressing the human language network using large language models

“…There are good reasons to adopt this approach: the different regions of this network i) have similar functional response profiles, both with respect to their selectivity for language (e.g., [13][14][15]17,18 ) and their responses to linguistic manipulations (e.g., 21,152 ), and ii) exhibit highly correlated time courses during naturalistic cognition paradigms (e.g., 80,82,153,127,154,11 ). However, some functional heterogeneity has been argued to exist within the language network (e.g., 30,29,32,151,155,156 ). Future efforts using an approach like the one adopted here could perhaps discover functional differences within the language network (by searching for stimuli that would selectively drive particular regions within the network), as well as between the core LH language network and the RH homotopic areas and other language-responsive cortical, subcortical, and cerebellar areas.…”

Neural populations in the language network differ in the size of their temporal receptive windows

“…Inspection of the average timecourse by cluster ( Figure 2E ) revealed three distinct response profiles (see Figure 2D for best representative electrodes from each cluster —’medoids’— chosen by the k-medoids algorithm). Cluster 1 (n=92 electrodes; range across participants: 5-34, Figure S1 ) is characterized by a relatively slow increase (build-up) of neural activity across the 8 words in the S condition (a pattern similar to the one reported by Fedorenko et al, 2016; Nelson et al, 2017; Desbordes et al, 2023; Woolnough et al, 2023; but see Discussion), and much lower activity for the W, J, and N conditions, with no difference between the J and N conditions ( Figure 2F ). Cluster 2 (n=67 electrodes; range across participants: 1-21, Figure S1 ) displays a quicker build-up of neural activity in the S condition that plateaus approximately 3 words into the sentence, a quick build-up of activity in the W condition that begins to decay after the third word, and a similar response to the J and N conditions as to the W condition with an overall lower magnitude.…”

“…We used intracranial recordings from patients with intractable epilepsy to investigate neural responses during language comprehension. Participants in Dataset 1 were presented with four types of linguistic stimuli that have been traditionally used to tease apart neural responses to word meanings and syntactic structure (Fedorenko et al, 2010(Fedorenko et al, , 2012(Fedorenko et al, , 2016Pallier et al, 2011;Shain, Kean et al, 2023;Desbordes et al, 2023; for earlier uses of this paradigm, see Mazoyer et al, 1993;Friederici et al, 2000;Humphries et al, 2001;Vandenberghe et al, 2002): sentences (S), lists of unconnected words (W), Jabberwocky sentences (J), and lists of unconnected nonwords (N) (Figure 1A,B, Methods, all stimuli are available at osf.io/xfbr8/). In each trial, 8 words or nonwords were presented on a screen serially and participants were asked to silently read them.…”

“…Using a similar design in an intracranial recording study, Fedorenko et al (2016) replicated this overall pattern of response, while additionally revealing its coarse temporal dynamics. More specifically, Fedorenko et al (2016) reported a temporal profile—present in a subset of electrodes—whereby high gamma power response builds-up over the course of a sentence but not in other conditions (replicated by Nelson et al, 2017; Desbordes et al, 2023; Woolnough et al, 2023), which they interpreted as indexing the process of constructing a sentence-level meaning. Here, we investigated the temporal profiles of language-responsive electrodes more comprehensively.…”

“…For example, Fedorenko et al (2016) reported sensitivity in language-responsive electrodes to both word meanings and combinatorial processing, in line with fMRI findings (e.g., Fedorenko et al, 2010; Bedny et al, 2011). They also reported a temporal profile where neural activity gradually increases (builds up) across the sentence (replicated by Nelson et al, 2017; Desbordes et al, 2023; Woolnough et al, 2023), which they interpreted as reflecting the construction of a sentence meaning. However, considerable disagreement exists in the field regarding the number of distinct profiles that characterize cortical language responses, how they functionally differ, and what computations they collectively support in the service of language comprehension and production.…”