Do we need to adjust for interim analyses in a Bayesian adaptive trial design?

Abstract

Background: Bayesian adaptive methods are increasingly being used to design clinical trials and offer several advantages over traditional approaches. Decisions at analysis points are usually based on the posterior distribution of the treatment effect. However, there is some confusion as to whether control of type I error is required for Bayesian designs as this is a frequentist concept.Methods: We discuss the arguments for and against adjusting for multiplicities in Bayesian trials with interim analyses. With two case studies we illustrate the effect of including interim analyses on type I/II error rates in Bayesian clinical trials where no adjustments for multiplicities are made. We propose several approaches to control type I error, and also alternative methods for decision-making in Bayesian clinical trials.Results: In both case studies we demonstrated that the type I error was inflated in the Bayesian adaptive designs through incorporation of interim analyses that allowed early stopping for efficacy and without adjustments to account for multiplicity. Incorporation of early stopping for efficacy also increased the power in some instances. An increase in the number of interim analyses that only allowed early stopping for futility decreased the type I error, but also decreased power. An increase in the number of interim analyses that allowed for either early stopping for efficacy or futility generally increased type I error and decreased power.Conclusions: Currently, regulators require demonstration of control of type I error for both frequentist and Bayesian adaptive designs, particularly for late-phase trials. To demonstrate control of type I error in Bayesian adaptive designs, adjustments to the stopping boundaries are usually required for designs that allow for early stopping for efficacy as the number of analyses increase. If the designs only allow for early stopping for futility then adjustments to the stopping boundaries are not needed to control type I error. If one instead uses a strict Bayesian approach, which is currently more accepted in the design and analysis of exploratory trials, then type I errors could be ignored and the designs could instead focus on the posterior probabilities of treatment effects of clinically-relevant values.

Hypothesis testing and confidence sets: why Bayesian not frequentist, and how to set a prior with a regulatory authority

“…Figure 49: Posterior probability that devices meet legal requirements as a function of the numbers of devices that are intact n good and broken n bad using the regulator's prior that is Beta (21,4) and the manufacturer's prior that is Beta(101,1). The top plot uses the regulator's prior alone, the bottom one the manufacturer's prior alone, and the middle one the proposed mixture prior.…”

“…Figure 50: Posterior probability that manufacturer's model and priors are correct as a function of the numbers of devices that are intact n good and broken n bad using the regulator's prior that is Beta (21,4) and the manufacturer's prior that is Beta(101,1). The starting point at (0, 0) has been marked as (0.2, 0.2) on these log scales.…”

“…In the author's experience, during the period 1980 -2020, frequentist methods have been used much more frequently than Bayesian ones in reported analyses, though this now is gradually changing. In particular, regulators such as the American Food and Drug Administration (FDA) and the European Medicines Agency (EMA) currently still insist on using frequentist methods for medical drug and device approvals testing ( [3,4]).…”

Optimizing a Bayesian Hierarchical Adaptive Platform Trial Design for Stroke Patients

“…In the STEP-STONE trial, multiplicity occurred since multiple patient populations were included and multiple interims analysis were performed. In practice, controlling of familywise type 1 error in Bayesian designs often relies on simulation (8,38,44,45) . In the STEP-STONE study, thresholds of parameters were determined to ensure overall type 1 error being controlled at 0.05 level through extensive simulation studies.…”

On Frequentist and Bayesian Sequential Clinical Trial Designs

“…On the other hand, Bayesian trial designs have become increasingly popular (Spiegelhalter et al, 1994;Berry, 2006;Berry et al, 2010). Insightful discussions on Bayesian sequential designs can be found in Cornfield (1966b), Berry (1985), Berry (1987), Jennison and Turnbull (1990), Freedman et al (1994), Emerson et al (2007), and Ryan et al (2020), among many others.…”