A common misapplication of statistical inference: Nuisance control with null-hypothesis significance tests

Abstract

Experimental research on behavior and cognition frequently rests on stimulus or subject selection where not all characteristics can be fully controlled, even when attempting strict matching. For example, when contrasting patients to controls, variables such as intelligence or socioeconomic status are often correlated with patient status. Similarly, when presenting word stimuli, variables such as word frequency are often correlated with primary variables of interest. One procedure very commonly employed to control for such nuisance effects is conducting inferential tests on confounding stimulus or subject characteristics. For example, if word length is not significantly different for two stimulus sets, they are considered as matched for word length. Such a test has high error rates and is conceptually misguided. It reflects a common misunderstanding of statistical tests: interpreting significance not to refer to inference about a particular population parameter, but about 1. the sample in question, 2. the practical relevance of a sample difference (so that a nonsignificant test is taken to indicate evidence for the absence of relevant differences). We show inferential testing for assessing nuisance effects to be inappropriate both pragmatically and philosophically, present a survey showing its high prevalence, and briefly discuss an alternative in the form of regression including nuisance variables.

Speech Planning at Turn Transitions in Dialog Is Associated With Increased Processing Load

“…The maximal random effects structures as justified by design which allowed models to converge were used (Barr, 2013;Barr, Levy, Scheepers, & Tily, 2013). A number of nuisance variables were included in the fixed effects structure of the models (Sassenhagen & Alday, 2016): the duration of the confederate turn, since the pre-recorded sentences differed in complexity; the number of objects to be named by the participant, since task difficulty increases with the number of choices (Hick, 1952); trial number, to account for changes over the course of the experiment; and a binary variable indicating whether the sentence structure of the confederate turn was reused in the response turn, since processing load might be influenced by structural priming (Pickering & Ferreira, 2008;Segaert, Menenti, Weber, Petersson, & Hagoort, 2012). The statistical significance of nuisance variables was not assessed.…”

Variability and Consistency in First and Second Language Processing: A Masked Morphological Priming Study on Prefixation and Suffixation

“…As explained by Sassenhagen and Alday (), performing inferential statistics such as t tests to verify the matching of items in different conditions is problematic for the following reasons. First, we would be making inferences about the specific items selected, which a t test (or the like) does not allow to make.…”

Structural Principles or Frequency of Use? An ERP Experiment on the Learnability of Consonant Clusters

“…In addition to the factors of interest, the phonetic parameters pitch (Hz), duration (ms), intensity (dB) were included as nuisance parameters on the basis of the results presented in 5.3.2 in order to control for possible confounds (cf. Sassenhagen and Alday, 2016). Since the full models are quite complex, their full summaries are in the Appendices 2, 3, while in the main text, we present selected Wald Type-II Chi-squared tests, which provide an ANOVA-like summary of effects.…”