Comprehensive benchmarking and ensemble approaches for metagenomic classifiers

Abstract

BackgroundOne of the main challenges in metagenomics is the identification of microorganisms in clinical and environmental samples. While an extensive and heterogeneous set of computational tools is available to classify microorganisms using whole-genome shotgun sequencing data, comprehensive comparisons of these methods are limited.ResultsIn this study, we use the largest-to-date set of laboratory-generated and simulated controls across 846 species to evaluate the performance of 11 metagenomic classifiers. Tools were characterized on the basis of their ability to identify taxa at the genus, species, and strain levels, quantify relative abundances of taxa, and classify individual reads to the species level. Strikingly, the number of species identified by the 11 tools can differ by over three orders of magnitude on the same datasets. Various strategies can ameliorate taxonomic misclassification, including abundance filtering, ensemble approaches, and tool intersection. Nevertheless, these strategies were often insufficient to completely eliminate false positives from environmental samples, which are especially important where they concern medically relevant species. Overall, pairing tools with different classification strategies (k-mer, alignment, marker) can combine their respective advantages.ConclusionsThis study provides positive and negative controls, titrated standards, and a guide for selecting tools for metagenomic analyses by comparing ranges of precision, accuracy, and recall. We show that proper experimental design and analysis parameters can reduce false positives, provide greater resolution of species in complex metagenomic samples, and improve the interpretation of results.Electronic supplementary materialThe online version of this article (doi:10.1186/s13059-017-1299-7) contains supplementary material, which is available to authorized users.

Clinical metagenomics

“…Well characterized reference standards and controls are needed to ensure mNGS assay quality and stability over time. Most available metagenomic reference materials are highly customized to specific applications (for exam ple, ZymoBIOMICS Microbial Community Standard for microbiome analyses and bacterial and fungal meta genomics 105 ) and/or focused on a more limited spec trum of organisms (for example, the National Institute of Standards and Technology (NIST) reference materials for mixed microbial DNA detection, which contain only bacteria 106 ). Thus, these materials may not be applicable to untargeted mNGS analyses.…”

“…Customized data sets can be prepared to mimic input sequence data and expand the range of microorganisms detected through in silico analysis 37 . The use of standardized reference mate rials and NGS data sets is also helpful in comparative evaluation of different bioinformatics pipelines 105 .…”

Keeping up with the genomes: efficient learning of our increasing knowledge of the tree of life

“…However, while many methods have been proposed for taxonomic classification [12,13], the accuracy of these methods using different training databases has not been fully tested. This is an important issue, because as new genome data are generated, training data sets, such as the commonly used the NCBI Reference Sequence Database (RefSeq) will change over time.…”

“…Taxonomic classification is usually one of the first steps in a metagenomic pipeline [11]. Once these organisms are identified, they are then used in downstream analyses, such as alpha/beta diversity measures, ordination, feature selection, phenotype classification, etc.However, while many methods have been proposed for taxonomic classification [12,13], the accuracy of these methods using different training databases has not been fully tested. This is an important issue, because as new genome data are generated, training data sets, such as the commonly used the NCBI Reference Sequence Database (RefSeq) will change over time.…”

BASTA – Taxonomic classification of sequences and sequence bins using last common ancestor estimations

“…Methods in Ecology and Evoluঞon KAHLKE And RALPH estimation implemented in the amplicon analysis framework QIIME (Caporaso et al, 2010) which uses the three best blast hits of a read for classification of amplicon sequences. Despite its reliance on well curated target databases, LCA algorithms have been shown to be highly accurate (McIntyre et al, 2017) and, depending on the comparison tool used, computationally efficient. Current LCA implementations, however, are restricted to NGS reads or short sequences and lack the ability to classify sequences such as predicted aminoacid sequences, assembled contigs from genome and metagenome projects or the increasingly common long-read sequences produced by 3GS technologies.…”