Optimal Multiple Testing and Design in Clinical Trials

Abstract

A central goal in designing clinical trials is to find the test that maximizes power (or equivalently minimizes required sample size) for finding a false null hypothesis subject to the constraint of type I error. When there is more than one test, such as in clinical trials with multiple endpoints, the issues of optimal design and optimal procedures become more complex. In this paper, we address the question of how such optimal tests should be defined and how they can be found. We review different notions of power and how they relate to study goals, and also consider the requirements of type I error control and the nature of the procedures. This leads us to an explicit optimization problem with objective and constraints that describe its specific desiderata. We present a complete solution for deriving optimal procedures for two hypotheses, which have desired monotonicity properties, and are computationally simple. For some of the optimization formulations this yields optimal procedures that are identical to existing procedures, such as Hommel's procedure or the procedure of Bittman et al. (2009), while for other cases it yields completely novel and more powerful procedures than existing ones. We demonstrate the nature of our novel procedures and their improved power extensively in a simulation and on the APEX study (Cohen et al., 2016).

Optimal and Maximin Procedures for Multiple Testing Problems

“…The first direction is to focus on applications to clinical trials, where a small number of tests K is typically performed and where optimizing power (equivalently, decreasing sample size to achieve equal power) can be especially critical for reducing costs. In follow‐up work (Heller et al, 2021), we have focused on the application to clinical trials, examining how multiple testing in clinical trials can be framed within our framework and demonstrating the potential benefits from following this route. In this framework, it is also possible to efficiently deal with general dependence structure and non‐exchangeability, as we discuss.…”

Discussion on “Optimal Test Procedures for Multiple Hypotheses Controlling the Familywise Expected Loss” by Willi Maurer, Frank Bretz, and Xiaolei Xun

“…However, the optimal decision rule among all weakly monotone rules that provide strong FWEL control is not typically a rectangular decision rule. For the binary loss function, we show in Heller et al (2022) that the optimal solution can be found using a simple and efficient algorithm. This solution is not necessarily within the class of rectangular decision rules.…”

“…Indeed, unless weights for losses are all equal, some hypotheses will be tested at a level higher than the FWEL level they should pass if tested individually. Obviously if we add a requirement that the individual level be constrained, so a hypothesis can be rejected only if its 𝑝-values are at most 𝛼 (the marginally nominal 𝛼 constraint suggested in Heller et al 2022), then for the separable rule the optimal rule is that rejection occurs for each hypothesis if and only if its 𝑝-value is ≤ 𝛼.…”

“…In contrast, if one chooses 𝜆 𝑖 = 𝑚, we get a weighted version of the PFER. For extremely unbalanced weights one can get rejection of hypotheses with 𝑝-values bigger than 𝛼, but these can be capped at 𝛼, as in Heller et al (2022), without limiting the flexibility. In Benjamini and Hochberg (1997), we introduced weights into the PFER, and proved the optimality theorem for controlling weighted PFER.…”

Rejoinder to Discussions on “Optimal Test Procedures for Multiple Hypotheses Controlling the Familywise Expected Loss”

“…In the specific example of Figure 2, the prevalence is different between the two subgroups, leading to larger thresholds for testing an effect in the smaller subgroup (and making it easier to reject the associated null hypothesis) because the costs of incorrect decisions are smaller. Of course, if one feels uncomfortable with rejection thresholds being larger than 𝛼, then one can include additional constraints, as suggested by Benjamini et al (2023) in reference to Heller et al (2022).…”