Comparisons of methods for multiple hypothesis testing in neuropsychological research.

Abstract

Hypothesis testing with multiple outcomes requires adjustments to control Type I error inflation, which reduces power to detect significant differences. Maintaining the prechosen Type I error level is challenging when outcomes are correlated. This problem concerns many research areas, including neuropsychological research in which multiple, interrelated assessment measures are common. Standard p value adjustment methods include Bonferroni-, Sidak-, and resampling-class methods. In this report, the authors aimed to develop a multiple hypothesis testing strategy to maximize power while controlling Type I error. The authors conducted a sensitivity analysis, using a neuropsychological dataset, to offer a relative comparison of the methods and a simulation study to compare the robustness of the methods with respect to varying patterns and magnitudes of correlation between outcomes. The results lead them to recommend the Hochberg and Hommel methods (step-up modifications of the Bonferroni method) for mildly correlated outcomes and the step-down minP method (a resampling-based method) for highly correlated outcomes. The authors note caveats regarding the implementation of these methods using available software. Neuropsychological datasets typically consist of multiple, partially overlapping measures, henceforth termed outcomes. A given neuropsychological domain-for example, executive function-is composed of multiple interrelated subfunctions, and frequently all subfunction outcomes of interest are subject to hypothesis testing. At a given α (critical threshold), the risk of incorrectly rejecting a null hypothesis, a Type I error, increases as more hypotheses are tested. This applies to all types of hypotheses, including a set of two-group comparisons across multiple outcomes (e.g., differences between two groups across several cognitive measures) or multiple-group comparisons within an analysis of variance framework (e.g., cognitive performance differences between several treatment groups and a control group). Collectively, we define these issues as the multiplicity problem (Pocock, 1997).Controlling Type I error at a desired level is a statistical challenge, further complicated by the correlated outcomes prevalent in neuropsychological data. By making adjustments to control Type I error, we increase the risk of incorrectly accepting a null hypothesis, a Type II error. In other words, we reduce power. Failure to control Type I error when examining multiple outcomes may yield false inferences, which may slow or sidetrack research progress. Researchers need strategies that maximize power while ensuring an acceptable Type I error rate.Many methods exist to manage the multiplicity problem. Several methods are based on the Bonferroni and Sidak inequalities (Sidak, 1967;Simes, 1986). These methods adjust α values or p values using simple functions of the number of tested hypotheses (Sankoh, Huque, & Dubey, 1997;Westfall & Young, 1993). Holm (1979), Hochberg (1988), and Hommel (1988 developed Bonferroni derivatives in...

Permeability of the blood–brain barrier predicts no evidence of disease activity at 2 years after natalizumab or fingolimod treatment in relapsing–remitting multiple sclerosis

“…Applying a Bonferroni correction but taking the correlation coefficient (CC; average CC = 0.49) between the measured variables into account by way of the Dubey–Armitage‐Parmar approach,30, 31 the threshold for rejecting the null hypothesis becomes p  = 0.024. All p values are thus reported uncorrected, but only described as significant if falling below p  = 0.024.…”

Monitoring Caribbean tobacco use: baseline adolescent smoking prevalence and regional disparities

“…With two types of prevalence rate among 14 countries for boys and girls, this creates 60 separate univariate tests of significance, and we control for the possibility of false positive statistical significance using the Šidák inequality. 16 Using this adjustment for multiple testing, we must achieve a p-value of 0.00085 or smaller to achieve statistical significance at the 5% level. Last, our four health disparity metrics were calculated: two absolute measures (PD, MD) and two relative measures (PR, ID) and reported separately for boys and girls.…”

Are Per-Family Type I Error Rates Relevant in Social and Behavioral Science?

“…Psychological researchers that have touted the superior power of stepwise methods over the Bonferroni procedure (e.g., Blakesley et al, 2009;Eichstaedt, Kovatch, and Maroof, 2013;Seaman, Levin, & Serlin, 1991) have rarely mentioned that such methods-though useful-do not control the PFER and therefore are not adequate substitutes for the Bonferroni procedure when control of the PFER is desired. For example, Eichstaedt and colleagues (2013, p. 693) explicitly stated, "The Holm's sequential procedure corrects for Type I error as effectively as the traditional Bonferroni method"-which is only true if the PFER is not considered (see Barnette & McLean, 2005).…”