Statistics in the Service of Science: Don’t Let the Tail Wag the Dog

Abstract

Statistical modeling is generally meant to describe patterns in data in service of the broader scientific goal of developing theories to explain those patterns. Statistical models support meaningful inferences when models are built so as to align parameters of the model with potential causal mechanisms and how they manifest in data. When statistical models are instead based on assumptions chosen by default, attempts to draw inferences can be uninformative or even paradoxical—in essence, the tail is trying to wag the dog. These issues are illustrated by van Doorn et al. (this issue) in the context of using Bayes Factors to identify effects and interactions in linear mixed models. We show that the problems identified in their applications (along with other problems identified here) can be circumvented by using priors over inherently meaningful units instead of default priors on standardized scales. This case study illustrates how researchers must directly engage with a number of substantive issues in order to support meaningful inferences, of which we highlight two: The first is the problem of coordination, which requires a researcher to specify how the theoretical constructs postulated by a model are functionally related to observable variables. The second is the problem of generalization, which requires a researcher to consider how a model may represent theoretical constructs shared across similar but non-identical situations, along with the fact that model comparison metrics like Bayes Factors do not directly address this form of generalization. For statistical modeling to serve the goals of science, models cannot be based on default assumptions, but should instead be based on an understanding of their coordination function and on how they represent causal mechanisms that may be expected to generalize to other related scenarios.

Bayes Factors for Mixed Models: Perspective on Responses

“…1) poorly specified and exclude them from consideration altogether. This stands in sharp contrast to both Heathcote and Matzke (2021), who argue against this exclusion, to Singmann et al (2021), who take the inclusion of these models as self-evident, and to Heck and Bockting (2021), who model-average across the model ensemble. Rouder et al (2021) build logically on the premise that Models 4 and 5 are not theoretically meaningful.…”

“…Below, we first discuss Rouder et al (2021), who support the principle of marginality; we then turn to Heathcote and Matzke (2021), who question the principle. This is followed by an extended discussion of Singmann et al (2021), who offer both a specific critique of assigning prior distributions to standardized effect sizes and a general critique of Bayes factors. The Singmann et al (2021) comment is echoed by Vasishth et al (2021), who also argue against aggregation, against omitting the intercept-slope correlations, and in favor of a sensitivity analysis.…”

“…This is followed by an extended discussion of Singmann et al (2021), who offer both a specific critique of assigning prior distributions to standardized effect sizes and a general critique of Bayes factors. The Singmann et al (2021) comment is echoed by Vasishth et al (2021), who also argue against aggregation, against omitting the intercept-slope correlations, and in favor of a sensitivity analysis. Heck and Bockting (2021) propose to model-average across the competing models, such that models that predict the data relatively well have an increased influence on the resulting inference.…”

“…In their commentary, Singmann et al (2021) argue that researchers need to think deeply about the link between theory and the statistical models they implement. This connection between theory and models also includes the choice of prior distributions.…”

“…The target article was followed by seven commentaries (i.e., Rouder et al, 2021;Heathcote and Matzke, 2021;Singmann et al, 2021;Vasishth et al, 2021;Heck & Bockting, 2021;Linde & van Ravenzwaaij, 2021; Johnny van Doorn and Frederik Aust contributed equally to this work.…”