From defaults to databases: parameter and database choice dramatically impact the performance of metagenomic taxonomic classification tools

Abstract

In metagenomic analyses of microbiomes, one of the first steps is usually the taxonomic classification of reads by comparison to a database of previously taxonomically classified genomes. While different studies comparing metagenomic taxonomic classification methods have determined that different tools are "best", there are two tools that have been used the most to-date: Kraken (k-mer based classification against a user-constructed database) and MetaPhlAn (classification by alignment to clade-specific marker genes), the latest versions of which are Kraken2 and MetaPhlAn 3, respectively. We found large discrepancies in both the proportion of reads that were classified as well as the number of species that were identified when we used both Kraken2 and MetaPhlAn 3 to classify reads within metagenomes from human-associated or environmental datasets. We then investigated which of these tools would give classifications closest to the real composition of metagenomic samples using a range of simulated and mock samples and examined the combined impact of tool-parameter-database choice on the taxonomic classifications given. This revealed that there may not be a one-size-fits-all "best" choice. While Kraken2 can achieve better overall performance, with higher precision, recall and F1 scores, as well as alpha- and beta-diversity measures closer to the known composition than MetaPhlAn 3, the computational resources required for this may be prohibitive for many researchers, and the default database and parameters should not be used. We therefore conclude that the best tool-parameter-database choice for a particular application depends on the scientific question of interest, which performance metric is most important for this question and the limit of available computational resources.

Phanta: Phage-inclusive profiling of human gut metagenomes

“…As viruses have relatively low abundance in a typical metagenomic sample, we chose to use whole genome classification, which is typically more sensitive in the low-coverage regime than methods relying on clade-specific marker genes 22,39 . Specifically, Phanta classifies reads to the lowest possible taxonomic rank by Kraken2 22,24 , a k -mer-based method that has been shown to be both fast and accurate given the correct database and optimized parameters 39 . Second, Phanta reduces false positive species by filtering out species based on a calculated proxy for genome coverage (see Methods), a known issue in taxonomic classification 40 .…”

“…Boxplots display the percentage distribution across the set of metagenomes. Database abbreviations: STD = standard Kraken2 44 , UHGG = Unified Human Gastrointestinal Genome Collection 18 , RefSeq = RefSeq Complete v205 39 , HumGut = HumGut 19 , Phanta = Phanta’s default database. The insert shows the same information as the boxplots for STD and Phanta.…”

“…These are the files necessary for running Kraken2 and Bracken. We obtained the number of prokaryotic and viral genomes in each database that we did not construct directly from the relevant publications: Wright et al ., 2022 (for Standard Kraken2 and RefSeq Complete) 39 ; Almeida et al ., 2021 (for UHGG) 18 ; Hiseni et al ., 2021 (for HumGut) 19 .…”

SPLASH: a statistical, reference-free genomic algorithm unifies biological discovery

“…Viral reads are often unaligned for similar reasons: viral reference genomes and reference transcriptomes cannot capture the complexity of viral quasispecies ( 19 ) or the vast extent of viral polymorphism and splicing ( 20 ), and new viral assemblies are constantly being discovered and added to reference databases ( 21, 22 ). It is impossible to imagine pre-specifying a set of reference genomes or transcriptomes due to the rapid genomic changes that define the microbial world and have significant clinical impact ( 23 ) and where the use of databases limits inference ( 24 ). In plant genomics and non-model organism work, it is common to lack a reference genome entirely, making inference on differential isoform expression through alignment impossible.…”

“…In the microbial world, pre-specifying a set of reference genomes is infeasible due to its inherent rapid genomic changes. References also cannot capture insertional diversity of mobile elements, which have significant phenotypic and clinical impact ( 11 ) and are only partially cataloged in references ( 12 ).…”