Evaluation of computational phage detection tools for metagenomic datasets

Abstract

IntroductionAs new computational tools for detecting phage in metagenomes are being rapidly developed, a critical need has emerged to develop systematic benchmarks.MethodsIn this study, we surveyed 19 metagenomic phage detection tools, 9 of which could be installed and run at scale. Those 9 tools were assessed on several benchmark challenges. Fragmented reference genomes are used to assess the effects of fragment length, low viral content, phage taxonomy, robustness to eukaryotic contamination, and computational resource usage. Simulated metagenomes are used to assess the effects of sequencing and assembly quality on the tool performances. Finally, real human gut metagenomes and viromes are used to assess the differences and similarities in the phage communities predicted by the tools.ResultsWe find that the various tools yield strikingly different results. Generally, tools that use a homology approach (VirSorter, MARVEL, viralVerify, VIBRANT, and VirSorter2) demonstrate low false positive rates and robustness to eukaryotic contamination. Conversely, tools that use a sequence composition approach (VirFinder, DeepVirFinder, Seeker), and MetaPhinder, have higher sensitivity, including to phages with less representation in reference databases. These differences led to widely differing predicted phage communities in human gut metagenomes, with nearly 80% of contigs being marked as phage by at least one tool and a maximum overlap of 38.8% between any two tools. While the results were more consistent among the tools on viromes, the differences in results were still significant, with a maximum overlap of 60.65%. Discussion: Importantly, the benchmark datasets developed in this study are publicly available and reusable to enable the future comparability of new tools developed.

Identification and Characterization of Novel Metagenome Assembled Uncultivated Virus Genomes from Human Gut

“…Notably, all four of these sequences were identified with the virus detection tools, VirFinder and DeepVirFinder, which are trained to identify prokaryotic viruses by k-mer based (deep)machine learning methods. Being homology-independent methods, they are known to have higher potential to identify novel sequences 37 . Among the 788 genomes that remained unaligned to the NCBI viral RefSeq database, 632 were complete genomes and 156 were >90% complete (Figure 1C).…”

Benchmarking Bioinformatic Virus Identification Tools Using Real-World Metagenomic Data across Biomes

“…Another one benchmarked tools that specialize in identifying viruses from clinical samples 51 . Four benchmarking works 49,50,52,53 mainly used simulated viral and non-viral testing datasets that were sampled from publicly available complete viral and microbial genomes (e.g., NCBI RefSeq). A summary of the tested tools and testing datasets of each study can be found in Supplementary Table S2.…”

“…Several benchmarking studies have compared the performance of various virus identification tools [48][49][50][51][52][53] (Supplementary Table S2). Most of them used simulated sequencing data or sequencing data from mock community as testing datasets.…”

“…Ho et al 53 benchmarked tools on a previously sequenced mock community containing five phage strains, which is low compared to real environmental samples. The newest benchmarking from Schackart et al 52 included two real-world metagenomic datasets but did not define a ground truth dataset.…”

ProkBERT Family: Genomic Language Models for Microbiome Applications

“…Current databases also show biases towards certain genera (Schackart III et al, 2023), which can skew benchmarking and the evaluation of different methods. To address this, we used a balanced benchmarking approach, ensuring each viral group corresponds to their predicted host genus, minimizing bias.…”