Optimal and Maximin Procedures for Multiple Testing Problems

Abstract

Multiple testing problems (MTPs) are a staple of modern statistical analysis. The fundamental objective of MTPs is to reject as many false null hypotheses as possible (that is, maximize some notion of power), subject to controlling an overall measure of false discovery, like family‐wise error rate (FWER) or false discovery rate (FDR). In this paper we provide generalizations to MTPs of the optimal Neyman‐Pearson test for a single hypothesis. We show that for simple hypotheses, for both FWER and FDR and relevant notions of power, finding the optimal multiple testing procedure can be formulated as infinite dimensional binary programs and can in principle be solved for any number of hypotheses. We also characterize maximin rules for complex alternatives, and demonstrate that such rules can be found in practice, leading to improved practical procedures compared to existing alternatives that guarantee strong error control on the entire parameter space. We demonstrate the usefulness of these novel rules for identifying which studies contain signal in numerical experiments as well as in application to clinical trials with multiple studies. In various settings, the increase in power from using optimal and maximin procedures can range from 15% to more than 100%.

Optimal Multiple Testing and Design in Clinical Trials

“…Although the connections to our previous work in Rosset et al (2022) are extensive, it is important to highlight the novel theoretical and methodological contributions of the current paper presented in Section 3. Most importantly, we relax the exchangeability requirement in Rosset et al (2022) and derive computationally efficient and conceptually simple algorithms. Both of these developments are enabled by our adoption of the weak monotonicity or marginally nominal-𝛼 requirement.…”

“…For practical problems, it may be useful to maximize the minimum power among all alternatives of interest, so the minimax objectives for a range of possible parameter settings 𝜗 ∈ Θ Π ⊂ Θ corresponding to Π 𝑎𝑛𝑦 , Π 𝑡𝑜𝑡𝑎𝑙 are, respectively (Romano et al, 2011;Rosenblum et al, 2014;Rosset et al, 2022):…”

“…For the case of two endpoints, a comprehensive comparison of the suggestion in Bittman et al (2009) with the Bonferroni and Simes local tests is given by Su et al (2012). Rosenblum et al (2014), and Rosset et al (2022) considered finding the optimal procedure for a power objective of interest while controlling in the strong sense a desired error rate, without restrictions to a selected set of allowed procedures. Determining the optimal multiple testing (OMT) procedure is computationally very challenging, so solutions were provided only for two hypothesis testing problems in Rosenblum et al (2014) and for two or three hypothesis testing problems in Rosset et al (2022).…”

“…Rosenblum et al (2014), and Rosset et al (2022) considered finding the optimal procedure for a power objective of interest while controlling in the strong sense a desired error rate, without restrictions to a selected set of allowed procedures. Determining the optimal multiple testing (OMT) procedure is computationally very challenging, so solutions were provided only for two hypothesis testing problems in Rosenblum et al (2014) and for two or three hypothesis testing problems in Rosset et al (2022). We find in this work that imposing restrictions of interest may not only result in a more attractive procedure, but also one that is computationally far simpler.…”

“…The OMT procedure for Π 𝑎𝑛𝑦 is to reject the smallest 𝑝-value if Φ −1 (𝑝 1 ) + Φ −1 (𝑝 2 ) ≤ √ 2Φ −1 (𝛼) (Rosenblum, 2014;Rosset et al, 2022). This procedure is not weakly monotone, since, for example, with realization 𝐩 = (𝛼∕2 + 𝛼 2 ∕4, 𝛼∕2 − 𝛼 2 ∕4) only the second hypothesis is rejected, and with realization 𝐪 = (𝛼∕3 − 𝛼 2 ∕4, 𝛼∕3 + 𝛼 2 ∕4) only the first hypothesis is rejected.…”