Indexing and searching petabyte-scale nucleotide resources

Abstract

Searching vast and rapidly growing sets of nucleotide content in data resources, such as runs in Sequence Read Archive and assemblies for whole genome shotgun sequencing projects in GenBank, is currently impractical in any reasonable amount of time or resources available to most researchers. We present Pebblescout, a tool that navigates such content by providing indexing and search capabilities. Indexing uses dense sampling of the sequences in the resource. Search finds subjects that have short sequence matches to a user query with well-defined guarantees. Reported subjects are ranked using a score that considers the informativeness of the matches. Six databases that index over 3.5 petabases were created and used to illustrate the functionality of Pebblescout. Here we show that Pebblescout provides new research opportunities and a data-driven way for finding relevant subsets of large nucleotide resources for analysis, some of which are missed when relying only on sample metadata or tools using pre-defined reference sequences. For two computationally intensive published studies, we show that Pebblescout rejects a significant number of runs analyzed without changing the conclusions of these studies and finds additional relevant runs. A pilot web service for interactively searching the six databases is freely available at https://pebblescout.ncbi.nlm.nih.gov/

DNA Polymerase Diversity Reveals Multiple Incursions of Polintons During Nematode Evolution

“…Polinton candidate sequences were further validated by the presence of at least 5 Polinton proteins through ORF prediction on candidate sequences and protein-protein BLAST (BLASTp, bitscore cutoff: 100) of the ORFs against query Polinton proteins. To ensure that the Polintons are from nematode (and not from other contaminating species), we scanned all NCBI-available genomes (WGS; all assemblies for the Whole Genome Shotgun sequencing projects available as of Feb 14, 2022) for closely matching sequences using the PebbleScout resource ( https://pebblescout.ncbi.nlm.nih.gov/ ) ( Shiryev and Agarwala 2023 ). The search across preindexed WGS nucleotide resources (17.33 terabases) detected only sequences from Nematoda genomes, arguing against any significant signal from contamination by unrelated species (this search does not rule out the possibility that other related nematodes have contaminated some assemblies; while unlikely, this situation would not substantially affect any of the conclusions of this work).…”

Viroid-like colonists of human microbiomes

“…Using Obelisk-ɑ as a starting point, 21 additional full-length examples of Obelisk-ɑ (<4 % sequence variation, Supplementary Table 1) were found in 7 datasets using a k-mer search ( 25 ) of ∼3.2 million “metagenomic” annotated sequence read archive (SRA) datasets. All 7 datasets were human-derived metatranscriptome (metagenomic RNA) BioProjects (Table 3, see Obelisk homologue detection in additional public data); 0 sequences were found in metagenomic DNA samples.…”

“…Close Obelisk-ɑ homologues were identified in the Short Read Archive (SRA) 52 using (“Metagenomic” database, default settings) 25 , a recently released tool that efficiently queries ∼3.2 million ( mid 2022 ) raw sequencing data for exact 42 k-mer matches. 9 metatranscriptome BioProjects (comprising 34 short read datasets) were identified ( > 65) with close (∼1 % nucleotide divergence) matches to Obelisk-ɑ, of which 3 were part of iHMP or its predecessor 71 , 5 were from other human stool studies 72–76 , and 1 was from a fox gut autopsy 77 .…”

A cryptic plastid and a novel mitochondrial plasmid inLeucomyxa plasmidiferagen. and sp. nov. (Ochrophyta) push the frontiers of organellar biology

“…The PRA-24 strain was isolated from a salt march in Virginia, but details on the original habitat and locality were not reported by Grant et al (2009) and are not available from the metadata provided by ATCC. To illuminate the ecological range of the respective organism, we extracted from its genome assembly the sequence of the highly variable rRNA ITS2 region and explored the occurrence of this “barcode” in raw metagenomic data using the PebbleScout search tool (Shiryev and Agarwala 2023). 461 metagenomic samples were retrieved matching the query with the maximal PBSscore value (i.e.…”