Large-scale benchmarking reveals false discoveries and count transformation sensitivity in 16S rRNA gene amplicon data analysis methods used in microbiome studies

Abstract

BackgroundThere is an immense scientific interest in the human microbiome and its effects on human physiology, health, and disease. A common approach for examining bacterial communities is high-throughput sequencing of 16S rRNA gene hypervariable regions, aggregating sequence-similar amplicons into operational taxonomic units (OTUs). Strategies for detecting differential relative abundance of OTUs between sample conditions include classical statistical approaches as well as a plethora of newer methods, many borrowing from the related field of RNA-seq analysis. This effort is complicated by unique data characteristics, including sparsity, sequencing depth variation, and nonconformity of read counts to theoretical distributions, which is often exacerbated by exploratory and/or unbalanced study designs. Here, we assess the robustness of available methods for (1) inference in differential relative abundance analysis and (2) beta-diversity-based sample separation, using a rigorous benchmarking framework based on large clinical 16S microbiome datasets from different sources.ResultsRunning more than 380,000 full differential relative abundance tests on real datasets with permuted case/control assignments and in silico-spiked OTUs, we identify large differences in method performance on a range of parameters, including false positive rates, sensitivity to sparsity and case/control balances, and spike-in retrieval rate. In large datasets, methods with the highest false positive rates also tend to have the best detection power. For beta-diversity-based sample separation, we show that library size normalization has very little effect and that the distance metric is the most important factor in terms of separation power.ConclusionsOur results, generalizable to datasets from different sequencing platforms, demonstrate how the choice of method considerably affects analysis outcome. Here, we give recommendations for tools that exhibit low false positive rates, have good retrieval power across effect sizes and case/control proportions, and have low sparsity bias. Result output from some commonly used methods should be interpreted with caution. We provide an easily extensible framework for benchmarking of new methods and future microbiome datasets.Electronic supplementary materialThe online version of this article (doi:10.1186/s40168-016-0208-8) contains supplementary material, which is available to authorized users.

Assessment of statistical methods from single cell, bulk RNA-seq and metagenomics applied to microbiome data

“…The similarities with respect to sparsity observed in both scRNA-seq and metagenomics data led us to pose the question of whether statistical methods developed for the differential expression of scRNA-seq data perform well on metagenomic DA analysis. Some benchmarking efforts have compared the performance of methods [9][10][11][12] both adapted from bulk RNA-seq and developed for microbiome DA 13,14 . While some tools exist to guide researchers 15 , a general consensus on the best approach is still missing, especially regarding the methods' capability of controlling false discoveries.…”

Dirichlet‐multinomial modelling outperforms alternatives for analysis of microbiome and other ecological count data

“…To quantify differences in proportions of features between two sampling groups [often referred to as ‘differential relative abundance testing’; Thorsen et al, , Weiss et al, ), posterior probability distributions (PPDs) for (Figure d) can be obtained. Consistent with convention, if 95% of the samples of this PPD of differences are either greater or less than zero, then there is a high certainty of a nonzero effect of sampling group on feature relative abundance.…”

“…To quantify differences in proportions of features between two sampling groups [often referred to as 'differential relative abundance testing'; Thorsen et al, 2016, Weiss et al, 2017, posterior probability distributions (PPDs) for j,k=1 − j,k=2 (Figure 2d) can be obtained.…”

“…Such effects will go unnoticed if analyses rely on techniques such as ordination and PERMANOVA, which can provide insight into overall differences between sampling groups (McKnight et al, 2019), but provide no statistical model to identify those features that may differ in relative abundance among groups. Accordingly, a variety of methods have been developed to perform the seemingly simple task of determining treatment-induced shifts in relative abundance, which is often referred to as 'differential relative abundance testing' or 'differential expression' testing (the latter phrase arises because the roots of many of these methods lie within the field of functional genomics; Bullard, Purdom, Hansen, & Dudoit, 2010;Dillies et al, 2013;Paulson, Stine, Bravo, & Pop, 2013;Thorsen et al, 2016;Weiss et al, 2017).…”

Effects of phosphorus-mobilizing bacteria on tomato growth and soil microbial activity

“…Operational taxonomic units (OTUs) responding significantly across experimental design were extracted using previously described methodology ) using an analysis of deviance (AOD) after generalized linear modelling (GLM) of the raw counts using negative binomial distribution (nb) with 1000 resampling iterations with residual variance, using the package mvabund (nbGLM, likelihood ratio test, p < 0.05, Wang et al 2012). This method was recently suggested as one of the most accurate way to extract significantly responding OTUs by minimizing the risk of error (Thorsen et al 2016). A generalized heatmap of dominant (relative abundance >0.1%) and significantly responding OTUs was generated using previously described methodology .…”