A practical guide to methods controlling false discoveries in computational biology

Abstract

Background: In high-throughput studies, hundreds to millions of hypotheses are typically tested. Statistical methods that control the false discovery rate (FDR) have emerged as popular and powerful tools for error rate control. While classic FDR methods use only p-values as input, more modern FDR methods have been shown to increase power by incorporating complementary information as "informative covariates" to prioritize, weight, and group hypotheses. However, there is currently no consensus on how the modern methods compare to one another. We investigated the accuracy, applicability, and ease of use of two classic and six modern FDR-controlling methods by performing a systematic benchmark comparison using simulation studies as well as six case studies in computational biology.Results: Methods that incorporate informative covariates were modestly more powerful than classic approaches, and did not underperform classic approaches, even when the covariate was completely uninformative. The majority of methods were successful at controlling the FDR, with the exception of two modern methods under certain settings. Furthermore, we found the improvement of the modern FDR methods over the classic methods increased with the informativeness of the covariate, total number of hypothesis tests, and proportion of truly non-null hypotheses.Conclusions: Modern FDR methods that use an informative covariate provide advantages over classic FDR-controlling procedures, with the relative gain dependent on the application and informativeness of available covariates. We present our findings as a practical guide and provide recommendations to aid researchers in their choice of methods to correct for false discoveries.

ideal: an R/Bioconductor package for Interactive Differential Expression Analysis

“…Further refinements to the results can be obtained by activating independent filtering (Bourgon et al, 2010), or selecting the more powerful Independent Hypothesis Weighting (IHW) framework (Ignatiadis et al, 2016), to ameliorate the multiple testing issue by incorporating an informative covariate, e.g. the mean gene expression (Korthauer et al, 2019). Shrinkage of the effect sizes is also optionally performed on the log fold change estimates, to reflect the higher levels of uncertainty for lowly expressed genes.…”

Towards a gold standard for benchmarking gene set enrichment analysis

“…On the other hand, the permutation procedure incorporated in many gene set tests has been shown to be biased [45], and inaccurate if permutation p-values are reported as zero [46]. Recent studies also reported nonuniform p-value distribution that are either systematically biased towards zero (false positive inflation) or one (false negative inflation) [47,48]. These shortcomings can lead to inappropriately small or large fractions of significant gene sets, and can considerably impair prioritization of gene sets in practice.…”

AdaFDR: a Fast, Powerful and Covariate-Adaptive Approach to Multiple Hypothesis Testing

“…We considered a subset of microbiome data from the Ecosystems and Networks Integrated with Genes and Molecular Assemblies (ENIGMA), where samples were acquired from monitoring wells in a site contaminated by former waste disposal ponds and all sampled wells have various geochemical and physical measurements 17,18 . Following the original study, we performed two experiments to test for correlations between the operational taxonomic units (OTUs) and the pH, Al respectively.…”

“…For the auditory experiment, the Brodmann areas corresponding to auditory cortices, namely 41,42,22, are among areas where the alternative hypotheses are most likely to occur. For the tennis imagination experiment, multiple cortices seem to respond to this stimulus, including auditory cortex (42), visual cortices (18,19), and motor cortices (4,6,7).…”

“…In all experiments, AdaFDR shows a similar improvement. Finally, we perform extensive simulations, including ones from a very recent benchmark paper (Oct 31st 2018) 18 , to demonstrate that AdaFDR has the highest detection power while controlling the false discovery proportion (FDP) in various cases where the p-values may be either independent or dependent. The default parameters of AdaFDR are used for every experiment in this paper, both real data analysis and simulations, without any tuning.…”