Discussion on “Optimal Test Procedures for Multiple Hypotheses Controlling the Familywise Expected Loss” by Willi Maurer, Frank Bretz, and Xiaolei Xun

Abstract

We discuss three issues. In the first part, we discuss the criteria emphasized by Maurer, Bretz, and Xun, warning that it modifies the per comparison error rate that does not address the concerns raised by multiple testing. In the second part, we strengthen the optimality results developed in the paper, based on our recent results. In the third part, we highlight the potentially important role that the use of weights may have in practice and discuss the difficulties in assigning weights that convey the importance in the gain and loss functions, especially as it pertains to multiple endpoints.

Rejoinder to Discussions on “Optimal Test Procedures for Multiple Hypotheses Controlling the Familywise Expected Loss”

“…We pointed out that the FWEL specializes to the conventional familywise error rate (FWER) when using binary step loss functions. Furthermore, the FWEL specializes to the per comparison error rate (PCER) and the per family error rate (PFER) when using additive step loss functions for specific choices of the losses 𝜆 𝑖 , as nicely illustrated by Benjamini et al (2023). They argued against using the PCER and referred to the decision rule in the lower left panel of Figure 2 in Maurer et al (2023) for potential consequences in terms of rejection thresholds becoming larger than 𝛼 if the tests had to be performed separately.…”

“…In the specific example of Figure 2, the prevalence is different between the two subgroups, leading to larger thresholds for testing an effect in the smaller subgroup (and making it easier to reject the associated null hypothesis) because the costs of incorrect decisions are smaller. Of course, if one feels uncomfortable with rejection thresholds being larger than 𝛼, then one can include additional constraints, as suggested by Benjamini et al (2023) in reference to Heller et al (2022).…”

“…Obviously, the loss induced by an incorrect full approval is larger than that of an incorrect conditional approval alone and the loss of both together is larger than each single one (Hsu et al, 2023). As pointed out by Benjamini et al (2023), weights have been introduced into the testing of multiple hypotheses a long while ago. There are particularly many applications of weighted tests to multiple endpoint analyses in clinical trials, where weights are typically selected for clinical reasons and not as the result of numerical optimization, all under the FWER framework (Westfall & Krishen, 2001;Bretz et al, 2009;Burman et al, 2009;Dmitrienko & Tamhane, 2013;Alosh et al, 2014).…”