Efficiency in Sequential Testing: Comparing the Sequential Probability Ratio Test and the Sequential Bayes Factor Test

Abstract

In a sequential hypothesis test, the analyst checks at multiple steps during data collectionwhether sufficient evidence has accrued to make a decision about the tested hypotheses.As soon as sufficient information has been obtained, data collection is terminated. Here,we compare two sequential hypothesis testing procedures that have recently been proposedfor use in psychological research: the Sequential Probability Ratio Test (SPRT; Schnuerch& Erdfelder, 2020) and the Sequential Bayes Factor Test (SBFT; Schönbrodt et al., 2017).We show that although the two methods have been presented as distinct methodologies inthe past, they share many similarities and can even be regarded as two instances of thesame overarching hypothesis testing framework. We demonstrate that the two methods usethe same mechanisms for evidence monitoring and error control, and that differences inefficiency between the methods depend on the exact specification of the statistical modelsinvolved. Given the close relationship between the SPRT and SBFT, we argue that thechoice of the sequential testing method should be regarded as a continuous choice withina unified framework rather than a dichotomous choice between two methods. We presentseveral considerations researchers can make to navigate the design decisions in the SPRTand SBFT.

Efficient alternatives for Bayesian hypothesis tests in psychology.

“…Schnuerch and Erdfelder (2020) do not provide objective criteria or default values for the standardized effect sizes that define the noncentrality parameters in these tests. Stefan et al (2021) discuss the connections between Schönbrodt et al (2017) and Schnuerch and Erdfelder (2020), pointing out that the thresholds for the Bayes factors in the former can be adjusted to control Type I and Type II error probabilities. Readers interested in more detailed descriptions of these and related sequential testing procedures are encouraged to consult Schönbrodt et al (2017) and Schnuerch and Erdfelder (2020).…”

“…Stefan et al (2021) point out that the SPRT can be modified for use with composite hypotheses by replacing the likelihood ratio with the Bayes factor between hypotheses. Hajnal (1961) and Schnuerch and Erdfelder (2020) extended the SPRT to t tests by replacing the likelihood ratio for normally distributed data with unknown means and common variance by the ratio of a noncentral t density to a central t density, evaluated at the t statistic for the experiment (e.g., t=nx¯/s).…”

Bayesian Sample Size Planning for Developmental Studies

“…However, it is also possible to conduct multiple BFDAs with different stopping rules in an iterative process. This allows researchers to plan a design that fulfills the requirements of a certain research scenario, for example, in terms of error control or evidence strength (Stefan, Schönbrodt, Evans, & Wagenmakers, 2022).…”

Model-Averaged Bayesian t-Tests

“…Unlike p-values, Bayes factors are consistent under both the null and the alternative hypothesis, meaning that as data accumulate indefinitely the chance that the Bayes factor points to the correct hypothesis approaches 1. This property enables hypothesis testers to stop whenever the evidence is deemed to be sufficiently compelling, and this allows for a flexible testing regime that is both efficient and ethical (Berger and Wolpert, 1988;Edwards et al, 1963;Rouder, 2014;Schönbrodt et al, 2017;Stefan, Schönbrodt, et al, 2020;Wagenmakers et al, 2022; for a discussion see de Heide and Grünwald, 2021;Hendriksen et al, 2021;Sanborn and Hills, 2014). 1 Third, the Bayesian framework allows researchers to quantify evidence in favor of the null hypothesis as well as the alternative hypothesis (Gallistel, 2009;Rouder et al, 2009).…”

“…Sequential analyses are also possible in the frequentist framework, although this usually entails an explicit advance commitment either to the number of interim tests or the precise value of the test-relevant parameter under the alternative hypothesis (see e.g.,Jennison and Turnbull, 1999;Lakens, 2014;Schnuerch and Erdfelder, 2020;Stefan, Schönbrodt, et al, 2020).…”