False positives and other statistical errors in standard analyses of eye movements in reading

Abstract

In research on eye movements in reading, it is common to analyze a number of canonical dependent measures to study how the effects of a manipulation unfold over time. Although this gives rise to the well-known multiple comparisons problem, i.e. an inflated probability that the null hypothesis is incorrectly rejected (Type I error), it is accepted standard practice not to apply any correction procedures. Instead, there appears to be a widespread belief that corrections are not necessary because the increase in false positives is too small to matter. To our knowledge, no formal argument has ever been presented to justify this assumption. Here, we report a computational investigation of this issue using Monte Carlo simulations. Our results show that, contrary to conventional wisdom, false positives are increased to unacceptable levels when no corrections are applied. Our simulations also show that counter-measures like the Bonferroni correction keep false positives in check while reducing statistical power only moderately. Hence, there is little reason why such corrections should not be made a standard requirement. Further, we discuss three statistical illusions that can arise when statistical power is low, and we show how power can be improved to prevent these illusions. In sum, our work renders a detailed picture of the various types of statistical errors than can occur in studies of reading behavior and we provide concrete guidance about how these errors can be avoided.

Incidental vocabulary learning in a natural reading context: an eye-tracking study

“…Each of these measures is believed to measure a different cognitive process (see Winke et al, 2013, for discussion). In this paper, we report the results for total time (i) because it is the variable that is of most pedagogical interest, (ii) because it has yielded the strongest associations with learning in previous studies, and (iii) because it encapsulates first fixation duration and gaze duration (see von der Malsburg & Angele, 2017, for caveats on multiple testing in eye-movement research). Additionally, we present scatter plots with fitted loess lines (i.e., smoothers) for all three words for first fixation duration, gaze duration, and total time in Appendix B.…”

Parafoveal preview effects from word N + 1 and word N + 2 during reading: A critical review and Bayesian meta-analysis

“…The problems associated with using multiple fixation duration measures have not been seriously considered until very Psychon Bull Rev (2017) 24:666-689 667 recently (von der Malsburg, & Angele, 2015). One issue with this practice is the high degree of collinearity between these measures.…”

“…In addition to them, second-pass reading measures are also sometimes used. The most common one is the total viewing time (TVT), which measures all fixations on the target word (including when the word is refixated during a regression).The problems associated with using multiple fixation duration measures have not been seriously considered until very Psychon Bull Rev (2017) 24:666-689 667 recently (von der Malsburg, & Angele, 2015). One issue with this practice is the high degree of collinearity between these measures.…”

Exploratory and Confirmatory Analyses in Sentence Processing: A Case Study of Number Interference in German

“…This approach has been criticized by statisticians because it introduces researcher degrees of freedom (Simmons et al, 2011). For a study exploring the consequences of Type I error inflation in eyetracking, see von der Malsburg and Angele (2017). Very similar problems occur in EEG and fMRI research.…”