An Analysis of the Use of Multiple Comparison Corrections in Ophthalmology Research

Abstract

PURPOSE. The probability of type I error, or a false-positive result, increases as the number of statistical comparisons in a study increases. Statisticians have developed numerous corrections to account for the multiple comparison problem. This study discusses recent guidelines involving multiple comparison corrections, calculates the prevalence of corrections in ophthalmic research, and estimates the corresponding number of false-positive results reported at a recent international research meeting.METHODS. The 6415 abstracts presented at ARVO 2010 were searched for statistical comparisons (P values) and for use of multiple comparison corrections. Studies that reported five or more P values while reporting no correction factor were used in a simulation study. The simulation study was conducted to estimate the number of false-positive results reported in these studies.RESULTS. Overall, 36% of abstracts reported P values and 1.2% of abstracts used some form of correction. Whereas 8% of abstracts reported at least five P values, only 5% of these used a multiple comparison correction. In these highly statistical studies, simulations resulted in 185 false-positive outcomes found in 30% of abstracts.CONCLUSIONS. The paucity of multiple comparison corrections in ophthalmic research results in inflated type I error and may produce unwarranted shifts in clinical or surgical care. Researchers must make a conscious effort to decide if and when to use a correction factor to ensure the validity of the data. (Invest Ophthalmol Vis Sci.

Workplace Psychosocial Factors Associated with Work-Related Injury Absence: A Study from a Nationally Representative Sample of Korean Workers

“…A lack of a comprehensive examination of interactions between a large number of the independent variables in the final models may have produced some random suppression/mediating effects of some variables on other variables. Type 1 error, a false-positive result, was also likely to occur because of the large set of comparisons [57, 58]. On the basis of the exploratory nature of the study, however, concerns about type 1 error associated with multiple comparisons may not be a serious problem with the preliminary study findings [59].…”

Correlations of inflammatory gene pathways, corticolimbic functional activities, and aggression in pediatric bipolar disorder: A preliminary study

“…Since this is an exploratory study, we did not implement multi-testing corrections to identify significant association findings. Prior studies have suggested that an inflated familywise error rate (FWER) due to a lack of multi-testing corrections may be acceptable in an exploratory study, since the hypothesis would need to be addressed a priori and cannot be discerned by any a posteriori analysis (Goeman and Solari, 2014; Stacey et al, 2012). …”

Effects of the KIBRA Single Nucleotide Polymorphism on Synaptic Plasticity and Memory: A Review of the Literature

“…The best practice to correct for these numerous statistical comparisons is by modifying the significance level (p-value) to account for the number of comparison made in the study [57]. When restricting the fMRI analysis to a brain region of interest, one accepted way to do this is using small volume correction (corresponding to the number of voxels compared) with the family wise error (FWE) rate (type I error rate) [58]. However, it is also important to consider that adopting the most conservative approach (FWE rate) may lead to increasing the rate of type II errors (false negatives) [59].…”