Critical Assessment of Metagenome Interpretation: the second round of challenges

Abstract

Evaluating metagenomic software is key for optimizing metagenome interpretation and focus of the Initiative for the Critical Assessment of Metagenome Interpretation (CAMI). The CAMI II challenge engaged the community to assess methods on realistic and complex datasets with long- and short-read sequences, created computationally from around 1,700 new and known genomes, as well as 600 new plasmids and viruses. Here we analyze 5,002 results by 76 program versions. Substantial improvements were seen in assembly, some due to long-read data. Related strains still were challenging for assembly and genome recovery through binning, as was assembly quality for the latter. Profilers markedly matured, with taxon profilers and binners excelling at higher bacterial ranks, but underperforming for viruses and Archaea. Clinical pathogen detection results revealed a need to improve reproducibility. Runtime and memory usage analyses identified efficient programs, including top performers with other metrics. The results identify challenges and guide researchers in selecting methods for analyses.

From defaults to databases: parameter and database choice dramatically impact the performance of metagenomic taxonomic classification tools

“…These previous studies have also used databases composed of only bacteria [1], only bacterial, archaeal, and viral genomes with complete assemblies [3], a MiniKraken database [1], or do not give full details on what is included in their database [2,4,5,7,9,17,18]. Additionally, while some other studies have used the NCBI non-redundant nucleotide database [6,8], we are not aware of any studies that have used the full NCBI RefSeq database that we have here, which is likely to have led to significantly worse performance in those previous comparisons (Fig. 4).…”

“…Due to the samples that were constructed by both the CAMI ( n =10) [5] and CAMI2 ( n =180) [6] studies using newly sequenced genomes, not all genomes – and therefore not all reads – have classifications at all taxonomic ranks. For some genomes, the lowest rank of the taxonomic classification given is at the family or genus level.…”

“…There are many metagenomic classification programs that have been developed to do this, and while these use a range of different methods, one common step is the comparison of unknown reads within a sample to a database of known genomes or sequences. Several studies have compared the performance of different metagenomic classifiers using simulated or mock communities with a known taxonomic composition [1][2][3][4] or a consensus approach [5,6] and publications introducing new metagenomic classifiers also often compare the method being introduced with previous methods [7][8][9]. The "best" metagenomic classifier is frequently determined based on either the F1 score (harmonic mean of precision and recall) or L1 distance (also known as Manhattan or Taxicab distance), and typically varies depending on the environment that a sample comes from as well as the complexity (number of taxa and magnitude of abundance differences) of the sample [1,8].…”

KMCP: accurate metagenomic profiling of both prokaryotic and viral populations by pseudo-mapping

“…In contrast, alignment-based methods show high tolerance for base variation. Marker gene approaches perform well in the identification of archaea and bacteria (Meyer et al ., 2022), while it is difficult to accurately identify viruses with this strategy since viruses do not have universally conserved genes, such as the 16S and 18S rRNA genes (Breitwieser et al ., 2019).…”

Persistent Memory as an Effective Alternative to Random Access Memory in Metagenome Assembly

“…These assemblers include meta-IDBA (Peng et al, 2011), metaSPAdes (Nurk et al, 2017), MEGAHIT (Li et al, 2016), and many others. Several recent studies have provided a comprehensive comparison of the computational performance and accuracy of these assemblers (Sczyrba et al, 2017; Vollmers et al, 2017; Meyer et al, 2021). While most of these assemblers can efficiently take advantage of the modern CPU’s multiple processing capabilities, they are limited on a single computer node and, therefore, are not able to assemble very large datasets due to the limited memory capacity.…”