We show that publishing results using the statistical significance filter-publishing only when the p-value is less than 0.05-leads to a vicious cycle of overoptimistic expectation of the replicability of results. First, we show analytically that when true statistical power is relatively low, computing power based on statistically significant results will lead to overestimates of power. Then, we present a case study using 10 experimental comparisons drawn from a recently published metaanalysis in psycholinguistics . We show that the statistically significant results yield an illusion of replicability. This illusion holds even if the researcher doesn't conduct any formal power analysis but just uses statistical significance to informally assess robustness (i.e., replicability) of results.

No abstract

A long-standing debate in the sentence processing literature concerns the time course of syntactic and semantic information in online sentence comprehension. The default assumption in cue-based models of parsing is that syntactic and semantic retrieval cues simultaneously guide dependency resolution. When retrieval cues match multiple items in memory, this leads to similarity-based interference. Both semantic and syntactic interferencehave been shown to occur in English. However, the relative timing of syntactic vs. semantic interference remains unclear. In this first-ever cross-linguistic investigation of the time course of syntactic vs. semantic interference, the data from two eye-tracking reading experiments (English and German) suggest that the two types of interference can in principle arise simultaneously during retrieval. However, the data also indicate that semantic cues may be evaluated with a small timing lag in German compared to English. This suggests that there may be cross-linguistic variation in how syntactic and semantic cues are used to resolve linguistic dependencies in real-time.

Cue-based parsing theories posit that dependency resolution during real-time sentence comprehension relies on cue-based retrieval of linguistic items encoded in memory. This retrieval mechanism is prone to similarity-based interference, which can occur when there are items in memory that are similar to the retrieval target. Interference during sentence comprehension seems to be well-established across numerous syntactic dependencies; however, the evidence for interference on within-sentence dependencies from sentence-external lexical items (encoded in memory prior to reading a target dependency) is inconclusive (Van Dyke &McElree, 2006; Van Dyke et al., 2014). The goal of the present study is to provide a large-scale cross-linguistic investigation of retrieval interference from sentence-external distractors under varying processing demands. Three larger-sample eye-tracking studies in English (N=66),German (N=122) and Russian (N=109) show no support for similarity-based interference from sentence-external material during sentence comprehension. We discuss the implications of our findings for cue-based parsing theories.

Preregistration is an open science practice that requires the specification of research hypotheses and analysis plans before the data are inspected. Here, we discuss the benefits of preregistration for hypothesis-driven, confirmatory bilingualism research. Using examples from psycholinguistics and bilingualism, we illustrate how non-peer reviewed preregistrations can serve to implement a clean distinction between hypothesis testing and data exploration. This distinction helps researchers avoid casting post-hoc hypotheses and analyses as confirmatory ones. We argue that, in keeping with current best practices in the experimental sciences, preregistration, along with sharing data and code, should be an integral part of hypothesis-driven bilingualism research.