Bayes factor design analysis: Planning for compelling evidence

Abstract

A sizeable literature exists on the use of frequentist power analysis in the null-hypothesis significance testing (NHST) paradigm to facilitate the design of informative experiments. In contrast, there is almost no literature that discusses the design of experiments when Bayes factors (BFs) are used as a measure of evidence. Here we explore Bayes Factor Design Analysis (BFDA) as a useful tool to design studies for maximum efficiency and informativeness. We elaborate on three possible BF designs, (a) a fixed-n design, (b) an open-ended Sequential Bayes Factor (SBF) design, where researchers can test after each participant and can stop data collection whenever there is strong evidence for either H 1 or H 0 , and (c) a modified SBF design that defines a maximal sample size where data collection is stopped regardless of the current state of evidence. We demonstrate how the properties of each design (i.e., expected strength of evidence, expected sample size, expected probability of misleading evidence, expected probability of weak evidence) can be evaluated using Monte Carlo simulations and equip researchers with the necessary information to compute their own Bayesian design analyses.

Introduction matters: Manipulating trust in automation and reliance in automated driving

“…For data analysis, we relied on Bayesian parameter estimation and Bayesian hypothesis tests by Bayes Factors (BF; (Körber, Radlmayr, & Bengler, 2016;Rouder, Speckman, Sun, Morey, & Iverson, 2009;Schönbrodt & Wagenmakers, 2017). A BF represents the ratio of the probability of the data given a null model to the probability of the data given an alternative model and thereby offers a gradual quantification of evidence (Schönbrodt, Wagenmakers, Zehetleitner, & Perugini, 2015).…”

Increasing propensity to mind‐wander by transcranial direct current stimulation? A registered report

“…Ethical approval for the study was granted at all three universities. Based on our design analysis (see below), we applied a sequential data collection protocol (Schönbrodt & Wagenmakers, 2018;Schönbrodt, Wagenmakers, Zehetleitner, & Perugini, 2017) and set out to collect between at least 120 and maximum 192 participants (a minimum of 20 and maximum of 32 participants per stimulation condition and study site). Subjects who failed to provide a complete dataset for technical (e.g., failure of the equipment) or other reasons (e.g., experiment not completed) were excluded from the analysis and replaced by new subjects.…”

“…In addition, we did not want to collect more data than necessary for ethical reasons. Therefore, we chose to apply a sequential design with a specified maximum sample size of N = 192 (Schönbrodt & Wagenmakers, 2018;Schönbrodt et al, 2017). In order to avoid spurious rejections of the existence of an effect, we chose to first collect a minimum sample size of N = 120 (20 per lab and condition).…”

Path Learning in Individuals With Down Syndrome: The Floor Matrix Task and the Role of Individual Visuo-Spatial Measures

“…A weakly informed Cauchy prior with rscale = √ 2/2 was used for H 1 (set by default by the "ttestBF" function). Referring to Schönbrodt and Wagenmakers (2018), we interpreted a BF > 3 as at least "moderate" evidence of H 1 (i.e., the hypothesis that the two means are not equal at the population level), and a BF < 1/3 as "moderate" evidence of H 0 (i.e., the hypothesis that the two means are equal at the population level). Any BF coming between these two cutoffs was regarded as "indecisive" evidence.…”