Large-scale replication study reveals a limit on probabilistic prediction in language comprehension

Abstract

Do people routinely pre-activate the meaning and even the phonological form of upcoming words? The most acclaimed evidence for phonological prediction comes from a 2005 Nature Neuroscience publication by DeLong, Urbach and Kutas, who observed a graded modulation of electrical brain potentials (N400) to nouns and preceding articles by the probability that people use a word to continue the sentence fragment (‘cloze’). In our direct replication study spanning 9 laboratories (N=334), pre-registered replication-analyses and exploratory Bayes factor analyses successfully replicated the noun-results but, crucially, not the article-results. Pre-registered single-trial analyses also yielded a statistically significant effect for the nouns but not the articles. Exploratory Bayesian single-trial analyses showed that the article-effect may be non-zero but is likely far smaller than originally reported and too small to observe without very large sample sizes. Our results do not support the view that readers routinely pre-activate the phonological form of predictable words.

Similar time courses for word form and meaning preactivation during sentence comprehension

“…Ito, Martin, and Nieuwland likewise obtained a marginally significant ( p = 0.06) prediction N400 effect at a/an articles for native English‐speaking participants (see discussion in DeLong, Urbach, & Kutas, ). An exception to these findings is a controversial multilab experiment by Nieuwland et al (), which reported a failed replication of the DeLong et al study (see Yan, Kuperberg, & Jaeger, , and a blog post by Shravan Vasishth, : https://vasishth-statistics.blogspot.com/2017/04/a-comment-on-delong-et-al-2005-nine.html for elaboration).…”

Frequency-specific brain dynamics related to prediction during language comprehension

“…Timings of theta-band patterns are consistent with known temporal characteristics of event-related electrophysiological responses (but see our remark on temporal estimates in section 2.6). The N400 response has been consistently linked to (predictive) aspects of lexical processing (Kutas & Federmeier, 2011, see also Nieuwland et al, 2018, for a criticial replication attempt). In sentence 630 reading, both and N400 effect and a spectral theta-band increase have been observed to unexpected words relative to expected ones (Rommers et al, 2017) suggesting that the two data analysis representations might reflect partially similar underlying biophysical processes.…”

Implicit semantic prediction error can account for N400 effects on articles that do not differ in meaning: A neural network model

“…The second issue is that one of the factors making it difficult to reliably observe article induced N400 effects in empirical studies is that the predictive relationship between indefinite articles and subsequent nouns is weakened by the fact that indefinite articles are not necessarily directly followed by nouns in natural language . Here we demonstrate the impact of the articles' cue validity by including simulations with very high versus very low cue validity, the assumption being that the cue validity in natural language presumably lies somewhere in between these extremes (according to the Corpus of Contemporary American English and British National Corpus, indefinite articles are directly followed by nouns in about a third of the cases; cited based on Nieuwland et al, 2018), thus contributing to the small size of the empirically observed effects (Nicenboim et al, 2019;Nieuwland et al, 2018).…”

“…Another issue concerning article induced N400 effects, which has recently been intensely debated, is that these effects have not always been replicated (for discussion see DeLong, Urbach, & Kutas, 2017;Ito, Martin, & Nieuwland, 2017b, 2017a, and in any case seem to be considerably smaller than observed in the original study (Nicenboim, Vasishth, & Rösler, 2019;Nieuwland et al, 2018). It has been suggested that this might be due to the fact that articles do not deterministically predict specific nouns in natural language (Ito et al, 2017b;Nieuwland et al, 2018). A second goal of the current study was to illustrate the importance of this factor, i.e.…”