Limits on prediction in language comprehension: A multi-lab failure to replicate evidence for probabilistic pre-activation of phonology

Abstract

In current theories of language comprehension, people routinely and implicitly predict upcoming words by pre-activating their meaning, morpho-syntactic features and even their specific phonological form. To date the strongest evidence for this latter form of linguistic prediction comes from a 2005 Nature Neuroscience landmark publication by DeLong, Urbach and Kutas, who observed a graded modulation of article- and noun-elicited electrical brain potentials (N400) by the pre-determined probability that people continue a sentence fragment with that word ('cloze'). In a direct replication study spanning 9 laboratories (N=334), we failed to replicate the crucial article-elicited N400 modulation by cloze, while we successfully replicated the commonly-reported noun-elicited N400 modulation. This pattern of failure and success was observed in a pre-registered replication analysis, a pre-registered single-trial analysis, and in exploratory Bayesian analyses. Our findings do not support a strong prediction view in which people routinely pre-activate the phonological form of upcoming words, and suggest a more limited role for prediction during language comprehension.

Prediction (or not) during language processing. A commentary on Nieuwland et al. (2017) and DeLong et al. (2005)

“…a larger negativity in association with higher article cloze probabilities, which was significant in two of the labs. One lab did reveal a correlation between the negativity evoked between 200-500ms and cloze probability in the expected direction, but this was significant at left frontal electrodes rather than at central parietal sites as found by DeLong et al (2005) (see Figure 1, Nieuwland et al, 2017).…”

“…In the first set of analyses-intended as a close replication of the correlation analyses performed In six of the labs, Nieuwland et al (2017) report a significant correlation between the noun's cloze probabilities and the N400 amplitude on the noun, with a similar classic N400 centroparietal topographic distribution as that reported by DeLong et al (2005); in two of the labs, the correlation trended, and in one lab it was insignificant (for further discussion of this, see DeLong et al, 2017a).…”

“…In the second set of analyses, Nieuwland et al (2017) go beyond the original analyses presented by DeLong et al, (2005) by analyzing the trial-level data. These analyses were conducted with linear mixed-effect regression, with cloze probability as a fixed-effect predictor and random byparticipant and by-item intercepts as well as slopes for cloze probability.…”

Why the A/AN prediction effect may be hard to replicate: A rebuttal to DeLong, Urbach & Kutas (2017)

“…We think this opposition is visible in psycholinguistics, too. Conceptual and direct replication research is unfortunately very sparse (but see Nieuwland et al, 2017;Rommers, Meyer, & Huettig, 2013;Zwaan & Pecher, 2012; see also Jäger, Engelmann, & Vasishth, 2017), and even novel but incremental contributions are often considered insufficient for publication.…”

“…Because Martin et al did not use cloze as a continuous predictor and because they used a different electrode-reference than ours, we based our prior on the output from a recent multi-lab replication attempt (Nieuwland et al, 2017; for discussion, see next section), which for a pre-registered analysis was an effect size of 0.296 µV for a 100% cloze difference (other aspects of the prior were the same as in Nieuwland et al). We here report the estimate of the effect size (b), the 95% Credible Interval (i.e., the interval of which we can be 95% confident contains the true effect), and the posterior probability of the estimated effect is positive or negative as a percentage.…”