Thinking ahead: spontaneous prediction in context as a keystone of language in humans and machines

Goldstein, Ariel; Zada, Zaid; Buchnik, Eliav; Schain, Mariano; Price, Amy; Aubrey, Bobbi; Nastase, Samuel A.; Feder, Amir; Emanuel, Dotan; Cohen, Alon; Jansen, Aren; Gazula, Harshvardhan; Choe, Gina; Rao, Aditi; Kim, Catherine; Casto, Colton; Fanda, Lora; Flinker, Adeen; Devore, Sasha; Doyle, Werner; Dugan, Patricia; Friedman, Daniel; Hassidim, Avinatan; Brenner, Michael P.; Matias, Yossi; Norman, Ken; Devinsky, Orrin; Hasson, Uri

doi:10.1101/2020.12.02.403477

Cited by 40 publications

(69 citation statements)

References 101 publications

(232 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, we found that the models’ performance on the next-word prediction task, but not other language tasks, is correlated with neural predictivity (see (Gauthier & Levy, 2019) for related evidence of fine-tuning of one model on tasks other than next-word-prediction leading to worse model-to-brain fit). Recent preprints conceptually replicate and extend this basic finding (Caucheteux & King, 2020; Goldstein et al, 2020; Wehbe et al, 2020; Wilcox et al, 2020). Language modeling (predicting the next word) is the task of choice in the natural language processing (NLP) community: it is simple, unsupervised, scalable, and appears to produce the most generally useful, successful language representations.…”

Section: Discussionmentioning

confidence: 69%

The neural architecture of language: Integrative modeling converges on predictive processing

Schrimpf

Blank²,

Tuckute

et al. 2020

Preprint

View full text Add to dashboard Cite

12brain predictivity, with each model's untrained score predictive of its trained score. These results support the hypothesis that a 13 drive to predict future inputs may shape human language processing, and perhaps the way knowledge of language is learned 14 and organized in the brain. In addition, the finding of strong correspondences between ANNs and human representations opens 15 the door to using the growing suite of tools for neural network interpretation to test hypotheses about the human mind. 16 computational neuroscience, language comprehension, fMRI, ECoG, human brain recordings, natural language processing, artificial neural networks, deep learning Specific models accurately predict human brain activity. We found ( Fig. 2a-b) that specific models predict Pereira2018 and 114Fedorenko2016 datasets with up to 100% predictivity (see Fig. S2 for generalization to another metric) relative to the noise 115 ceiling (Methods-7, Fig. S1). The Blank2014 dataset is also reliably predicted, but with lower predictivity. Models vary 116 substantially in their ability to predict neural data. Generally, embedding models such as GloVe do not perform well on any 117 dataset. In contrast, recurrent networks such as skip-thoughts, as well as transformers such as BERT, predict large portions 118 of the data. The model that predicts the human data best across datasets is GPT2-xl, which predicts Pereira2018 and 119Fedorenko2016 at close to 100% and is among the highest-performing models on Blank2014 with 32% predictivity. These 120 scores are higher in the language network than other parts of the brain (SI-4). 121Model scores are consistent across experiments/datasets. To test the generality of the model representations, we examined the 122 consistency of model scores across datasets. Indeed, if a model does well on one dataset, it tends to also do well on other 123 datasets ( Fig. 2c), ruling out the possibility that we are picking up on spurious, dataset-idiosyncratic predictivity, and 124suggesting that the models' internal representations are general enough to capture brain responses to diverse linguistic 125 materials presented visually or auditorily, and across three independent sets of participants. Specifically, model scores 126across the two experiments in Pereira2018 (overlapping sets of participants) correlate at r=.94 (Pearson here and 127 elsewhere, p<<.00001), scores from Pereira2018 and Fedorenko2016 correlate at r=.50 (p<.001), and from Pereira2018 and 128Blank2014 at r=.63 (p<.0001). 129 130Next-word-prediction task performance predicts neural scores. In vision, ANNs that perform better on the specific task of 131 visual classification also tend to better predict responses in the primate ventral stream (Schrimpf et al., 2018; Yamins et al., 132 2014). Building on work that has established a core role for predictive processing in language (Hale, 2001; Levy, 2008a; 133 Smith & Levy, 2013) and recent findings that ANNs that perform well on a next-word prediction task (a normative task 134 known as 'languag...

show abstract

Section: Discussionmentioning

confidence: 69%

The neural architecture of language: Integrative modeling converges on predictive processing

Schrimpf

Blank²,

Tuckute

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Crucially for us, the next-word probability estimates produced by these models was shown to be an impressive predictor of a host of indices of behavioral performance and brain activation, surpassing preceding generations of neural network models (reading times: Merkx & Frank, 2020;Wilcox et al, 2020;fMRI data: Schrimpf et al, 2020;MEG data: Caucheteux & King, 2020;N400 amplitudes: Merkx & Frank, 2020;Heilbron et al, 2021;ECOG activity: Goldstein et al, 2020;Schrimpf et al, 2020). Importantly, the improvement in linguistic accuracy (i.e., how good a model is at predicting the next word) converges with the model's "psychological" accuracy, i.e., how well it predicts context-based facilitations as measured in reading behavior and in the brain (Caucheteux & King, 2020;Goldstein et al, 2020;Schrimpf et al, 2020). Moreover, it is particularly the model's accuracy in the next-word prediction task (and not any other index of the model's performance) that explains its success in modeling brain activity (Schrimpf et al, 2020), which may indicate that-at least at the computational level-the brain pursues the same objective as the models trained in the next-word prediction task.…”

Section: Current Studymentioning

confidence: 99%

Context-Based Facilitation of Semantic Access Follows Both Logarithmic and Linear Functions of Stimulus Probability

Szewczyk¹,

Federmeier²

2021

Preprint

View full text Add to dashboard Cite

Stimuli are easier to process when the preceding context (e.g., a sentence, in the case of a word) makes them predictable. However, it remains unclear whether context-based facilitation arises due to predictive preactivation of a limited set of relatively probable upcoming stimuli (with facilitation then linearly related to probability) or, instead, arises because the system maintains and updates a probability distribution across all items, as posited by accounts (e.g., surprisal theory) assuming a logarithmic function between predictability and processing effort. To adjudicate between these accounts, we measured the N400 component, an index of semantic access, evoked by sentence-final words of varying probability, including unpredictable words, which are never generated in human production norms. Word predictability was measured using both cloze probabilities and a state-of-the-art machine learning language model (GPT-2). We reanalyzed five datasets (n=138) to first demonstrate and then replicate that context-based facilitation on the N400 is graded and dissociates even among words with cloze probabilities at or near 0, as a function of very small differences in model-estimated predictability. Furthermore, we established that the relationship between word predictability and context-based facilitation on the N400 is neither purely linear nor purely logarithmic but instead combines both functions. We argue that such a composite function reveals properties of the mapping between words and semantic features and how feature- and word- related information is activated during on-line processing. Overall, the results provide powerful evidence for the role of internal models in shaping how the brain apprehends incoming stimulus information.

show abstract

“…Moreover, the improvement of dynamical connectivity will also benefit the study of effective connectivity, causal relations between temporal signals, which should also greatly help clarify the "how" of language memory operations are performed by the brain and their failure. Finally, computational methods and artificial neural networks are also promising tools that, coupled with functional brain markers, can allow a better understanding of the computations and algorithms involved in natural language processing as well as their neural implementation (e.g., Goldstein et al, 2021;Jain & Huth, 2018, for two examples on word prediction in natural context; Martin, 2020, for a proposed architecture of the hierarchical and compositional structure of language, based on neurobiological and neurocomputational modeling evidence).…”

Section: Leveraging Advances In Functional Connectomicsmentioning

confidence: 99%

Missing links: the functional unification of language and memory (L∪M)

Roger¹,

Banjac²,

Schotten³

et al. 2021

Preprint

View full text Add to dashboard Cite

The field of neurocognition is currently undergoing a significant change of perspective. Traditional neurocognitive models evolved into an integrative and dynamic vision of cognitive functioning. Dynamic integration assumes an interaction between cognitive domains traditionally considered to be distinct. Language and declarative memory are regarded as separate functions supported by different neural systems. However, they also share anatomical structures (notably, the inferior frontal gyrus, the supplementary motor area, the superior and middle temporal gyrus, and the hippocampal complex) and cognitive processes (such as semantic and working memory) that merge to endorse our quintessential daily lives. We propose a new model, "L∪M" (i.e., Language/union/Memory), that considers these two functions interactively. We fractionated language and declarative memory into three cognitive dimensions, Embodiment-Formulation-Internalization, that communicate reciprocally. We formalized their interactions at the brain level with a connectivity-based approach. This new taxonomy overcomes the modular view of cognitive functioning and reconciles functional specialization with plasticity in neurological disorders.

show abstract

Thinking ahead: spontaneous prediction in context as a keystone of language in humans and machines

Cited by 40 publications

References 101 publications

The neural architecture of language: Integrative modeling converges on predictive processing

The neural architecture of language: Integrative modeling converges on predictive processing

Context-Based Facilitation of Semantic Access Follows Both Logarithmic and Linear Functions of Stimulus Probability

Missing links: the functional unification of language and memory (L∪M)

Contact Info

Product

Resources

About