Workflow techniques for the robust use of bayes factors.

Abstract

Inferences about hypotheses are ubiquitous in the cognitive sciences. Bayes factors provide one general way to compare different hypotheses by their compatibility with the observed data. Those quantifications can then also be used to choose between hypotheses. While Bayes factors provide an immediate approach to hypothesis testing, they are highly sensitive to details of the data/model assumptions and it's unclear whether the details of the computational implementation (such as bridge sampling) are unbiased for complex analyses. Here, we study how Bayes factors misbehave under different conditions. This includes a study of errors in the estimation of Bayes factors; the first-ever use of simulation-based calibration to test the accuracy and bias of Bayes factor estimates using bridge sampling; a study of the stability of Bayes factors against different MCMC draws and sampling variation in the data; and a look at the variability of decisions based on Bayes factors using a utility function. We outline a Bayes factor workflow that researchers can use to study whether Bayes factors are robust for their individual analysis. Reproducible code is available from https://osf.io/y354c/.

The power of “good”: Can adjectives rapidly decrease as well as increase the availability of the upcoming noun?

“…To estimate if a model assuming a D KL effect should be preferred over the null model that does not assume a D KL effect, we used Bayes Factors and sensitivity analysis. Sensitivity analysis addresses a potential limitation of Bayes factors, namely that, unlike posterior estimates, they critically depend on the priors and that there is little information on how these priors should be set given that we do not even know if the tested effect is different from zero (see Schad et al, 2021 for discussion; for an example of use, see Nicenboim et al, 2020). Sensitivity analysis consists in testing different priors and checking how their choice affects the Bayes factor value.…”

Data aggregation can lead to biased inferences in Bayesian linear mixed models and Bayesian ANOVA: A simulation study

“…Under these conditions, non-aggregated data analysis is needed for accurate and correct null hypothesis Bayes factor estimation. In the present work, we perform SBC for Bayes factors (Schad et al, 2022) to illustrate and demonstrate these points.…”

“…One limitation of prior research on Bayes factors has been that it was often unclear whether a Bayes factor estimate from a complex LMM is accurate, i.e., whether it corresponds to the true Bayes factor, making investigation of potential biases associated with data aggregation difficult. Recently, we have developed simulation-based calibration (SBC) for Bayes factors (Schad et al, 2022) (also see Cook, Gelman, & Rubin, 2006;Schad, Betancourt, & Vasishth, 2021;Talts, Betancourt, Simpson, Vehtari, & Gelman, 2018) that allows us to test the accuracy of Bayes factor estimates, i.e., whether the estimates are correct and correspond to the true value. Here, we use these new developments and apply them to the question of data aggregation in Bayesian analyses.…”

“…Importantly, this problem, that fitting LMMs on aggregated data yields biased results extends even to Bayes factor based analyses, when the sphericity assumption is violated. Here, we will illustrate this using one exemplary simulated data set, and perform simulation-based calibration (SBC) analyses DATA AGGREGATION (Schad et al, 2022)…”

“…Bayes factors can be used to test the null hypothesis that some effect is absent (i.e., that a parameter is zero) against an alternative hypothesis that the effect exists (i.e., that a parameter is different from zero), where Bayes factors perform the test under some prior assumed effect size. Bayes factor null hypothesis tests arguably provide a better alternative to frequentist p-values (Jeffreys, 1939;Kass & Raftery, 1995;Oberauer, 2022;Rouder, Haaf, & Vandekerckhove, 2018;Schad, Nicenboim, Bürkner, Betancourt, & Vasishth, 2022;Tendeiro & Kiers, 2019van Doorn, Aust, Haaf, Stefan, & Wagenmakers, 2021;van Ravenzwaaij & Wagenmakers, 2021;Wagenmakers, Lodewyckx, Kuriyal, & Grasman, 2010). In recent years, software has been developed that allows easy access to Bayesian hypothesis testing for lay users, such as the Bayesian analysis software WinBUGS (Lunn, Thomas, Best, & Spiegelhalter, 2000), JAGS (Plummer, 2003), PyMC3 (Salvatier, Wiecki, & Fonnesbeck, 2016), Stan (Carpenter et al, 2017), Turing (Ge, Xu, & Ghahramani, 2018), JASP (JASP Team, 2022), and others.…”