Balancing Type I error and power in linear mixed models

Abstract

Linear mixed-effects models have increasingly replaced mixed-model analyses of variance for statistical inference in factorial psycholinguistic experiments. Although LMMs have many advantages over ANOVA, like ANOVAs, setting them up for data analysis also requires some care. One simple option, when numerically possible, is to fit the full variance-covariance structure of random effects (the maximal model; Barr, Levy, Scheepers & Tily, 2013), presumably to keep Type I error down to the nominal α in the presence of random effects. Although it is true that fitting a model with only random intercepts may lead to higher Type I error, fitting a maximal model also has a cost: it can lead to a significant loss of power. We demonstrate this with simulations and suggest that for typical psychological and psycholinguistic data, higher power is achieved without inflating Type I error rate if a model selection criterion is used to select a random effect structure that is supported by the data.

The cave of shadows: Addressing the human factor with generalized additive mixed models

“…Would researchers come to 'wrong' conclusions if they analyze data simply using maximal linear models, without taking the human factor into account, without paying attention to whether the model is overfitting the data with mathematically uninterpretable parameters (Lele et al, 2012;Bates et al, 2015), and accepting less than nominal power (Matuschek et al, 2016)? The problem here is that low-hanging fruit is easily plucked, often by simple linear models without any random effects.…”

“…Since the proposed standard of Barr et al is overly conservative with an unnecessary loss of power (Matuschek et al, 2016), and since it comes with the risk of basing conclusions on mathematically ill-defined models that overfit the data (Lele et al, 2012;Bates et al, 2015), Barr et al's proposal has the unfortunate side-effect of locking analytical practice in a methodological cage of shadows from which the true structure of experimental data, rich and fertile in perhaps unexpected ways, cannot be fully appreciated.…”

Bayesian linear mixed models using Stan: A tutorial for psychologists, linguists, and cognitive scientists

“…Fitting a large number of random effects in non-Bayesian settings requires a large amount of data. Often, the data-set is too small to reliably estimate variance component parameters (Bates, Kliegl, et al, 2015;Matuschek, Kliegl, Vasishth, Baayen, & Bates, 2016). However, if a researcher is interested in differences between individual subjects or items (random intercepts and random slopes) or relationships between differences (correlations between variance components), Bayesian modeling can be used even if there is not enough data for inferential statistics.…”

“…Furthermore, as Barr et al (2013) point out, it is in principle desirable to include a fixed effect factor in the random effects as a varying slope if the experiment design is such that subjects see both levels of the factor (cf. Bates, Kliegl, et al, 2015;Matuschek et al, 2016;Baayen, Vasishth, Bates, & Kliegl, 2016).…”

Predictive language processing revealing usage-based variation

“…Furthermore, there are items no one had mentioned in the Completion task, thus prohibiting by-item random slopes for Ta r g e t M e n t i o n e d. Within these limits, a model with a full random effect structure was constructed following Barr et al (2013). Subsequently, we excluded random slopes with the lowest variance step by step until a further reduction would imply a significant loss in the goodness of fit of the model (Matuschek, Kliegl, Vasishth, Baayen, & Bates, 2017). Model comparisons indicated that the inclusion of the by-participant random slopes for Wo r d L e n g t h , P r e s e n tat i o n O r d e r , C l o z e P r o b a b i l i ty, and Ta r g e t M e n t i o n e d was justified by the data (χ2(5) = 53.00, p < .001).…”