Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models generate appropriate linguistic responses in a given context. In the current study, nine participants listened to a 30-min podcast while their brain responses were recorded using electrocorticography (ECoG). We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational principles as they process the same natural narrative: (1) both are engaged in continuous next-word prediction before word onset; (2) both match their pre-onset predictions to the incoming word to calculate post-onset surprise; (3) both rely on contextual embeddings to represent words in natural contexts. Together, our findings suggest that autoregressive DLMs provide a new and biologically feasible computational framework for studying the neural basis of language.

Departing from rule-based linguistic models, advances in deep learning resulted in a new type of autoregressive deep language models (DLMs). These models are trained using a self-supervised next word prediction task. We provide empirical evidence for the connection between autoregressive DLMs and the human language faculty using spoken narrative and electrocorticographic recordings. Behaviorally, we demonstrate that humans have a remarkable capacity for word prediction in natural contexts, and that, given a sufficient context window, DLMs can attain human-level prediction performance. Leveraging on DLM embeddings we demonstrate that the brain constantly and spontaneously predicts the identity of the next word in natural speech, hundreds of milliseconds before they are perceived. Finally, we demonstrate that contextual embeddings derived from autoregressive DLMs capture neural representations of the unique, context-specific meaning of words in the narrative. Our findings suggest that DLMs provides a novel biologically feasible computational framework for studying the neural basis of language.

A left-lateralized network of frontal and temporal brain regions is specialized for language processing-spoken, written, or signed. Different regions of this 'language network' have all been shown to be sensitive to various forms of linguistic information, from combinatorial sentence structure to word meanings, to sub-lexical regularities. However, whether neural computations are the same across and within these different brain regions remains debated. Here, we examine responses during language processing recorded intracranially in patients with intractable epilepsy. Across two datasets (Dataset 1: n=6 participants, m=177 language-responsive electrodes; Dataset 2: n=16 participants, m=362 language-responsive electrodes), we clustered language-responsive electrodes and found three distinct response profiles, with differences in response magnitude between linguistic conditions (e.g., sentences vs. lists of words), different temporal dynamics over the course of the stimulus, and different degrees of stimulus locking. We argue that these profiles correspond to different temporal receptive windows that vary in size between sub-lexical units and multi-word sequences. These results demonstrate the functional heterogeneity of neural responses in the language network and highlight the diversity of neural computations that may be needed in order to extract meaning from linguistic input. Importantly, electrodes that exhibit these distinct profiles do not cluster spatially and are instead interleaved across frontal and temporal language areas, which likely made It difficult to uncover functional differences in past fMRI studies. This mosaic of neural responses across the language network suggests that all language regions have direct access to distinct response types-a property that may be crucial for the efficiency and robustness of language processing mechanisms.