Squeakr: an exact and approximate <i>k</i>-mer counting system

Pandey, Prashant; Bender, Michael A.; Johnson, Rob; Patro, Rob

doi:10.1093/bioinformatics/btx636

Cited by 70 publications

(84 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also used BLight to associate to each k-mer its number of occurrences across the datasets to show a straightforward proof of concept application of the library. We selected a dataset from TARA Oceans samples [29], and compare BLight to two lightweight recent k-mer abundance indexes: Short Read Connector (SRC) [14] and Squeakr [30]. Lastly we show that our index can also be used on other SPSS than the unitigs of the compacted de Bruijn graph.…”

Section: Resultsmentioning

confidence: 99%

“…We compare a simple usage of BLight through a k-mer counting snippet, with two methods from state-ofthe-art that allow large scale k-mer to abundance association. Squeakr [33] is a k-mer counter based on a quotienting hashing technique, and Short Read Connector counter (SRC) [14] is based on a MPHF. We report the performances of the three tools on a large marine metagenomic dataset from TARA used the previous metagenomic experiment (ERR599280), counting 37 billion bases and 189 million reads.…”

Section: Application Example: Storing K-mer Countsmentioning

confidence: 99%

See 1 more Smart Citation

Efficient exact associative structure for sequencing data

Marchet

Kerbiriou

Limasset

2019

Preprint

View full text Add to dashboard Cite

Motivation: A plethora of methods and applications share the fundamental need to associate information to words for high throughput sequence analysis. Indexing billions of k-mers is promptly a scalability problem, as exact associative indexes can be memory expensive. Recent works take advantage of the properties of the k-mer sets to leverage this challenge. They exploit the overlaps shared among k-mers by using a de Bruijn graph as a compact k-mer set to provide lightweight structures. Results: We present Blight, a static and exact index structure able to associate unique identifiers to indexed k-mers and to reject alien k-mers that scales to the largest kmer sets with a low memory cost. The proposed index combines an extremely compact representation along with very high throughput. Besides, its construction from the de Bruijn graph sequences is efficient and does not need supplementary memory. The efficient index implementation achieves to index the k-mers from the human genome with 8GB within 10 minutes and can scale up to the large axolotl genome with 63 GB within 76 minutes. Furthermore, while being memory efficient, the index allows above a million queries per second on a single CPU in our experiments, and the use of multiple cores raises its throughput. Finally, we also present how the index can practically represent metagenomic and transcriptomic sequencing data to highlight its wide applicative range. Availability: The index is implemented as a C++ library, is open source under AGPL3 license, and available at github.com/Malfoy/Blight. It is designed as a user-friendly library and comes along with samples code usage.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Application Example: Storing K-mer Countsmentioning

confidence: 99%

Efficient exact associative structure for sequencing data

Marchet

Kerbiriou

Limasset

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…The compression of k-mer sets has not been extensively studied, except in the context of how k-mer counters store their output [17][18][19][20]. DSK [18] uses an HDF5-based encoding, KMC3 [17] combines a dense storage of prefixes with a sparse storage of suffixes, and Squeakr [20] uses a counting quotient filter [21]. The compression of read data, on the other hand, stored in either unaligned or aligned formats, has received a lot of attention [22][23][24].…”

Section: Related Workmentioning

confidence: 99%

“…All files are compressed with MFC or LZMA, in addition to the tool shown in the column name. Squeakr-exact's implementation is limited to k < 32 [20] and so it could not be run for k = 61.…”

Section: Evaluation Of Ust-fmmentioning

confidence: 99%

Representation of k-mer sets using spectrum-preserving string sets

Rahman

Medvedev

2020

Preprint

View full text Add to dashboard Cite

Given the popularity and elegance of k-mer based tools, finding a space-efficient way to represent a set of k-mers is important for improving the scalability of bioinformatics analyses. One popular approach is to convert the set of k-mers into the more compact set of unitigs. We generalize this approach and formulate it as the problem of finding a smallest spectrum-preserving string set (SPSS) representation. We show that this problem is equivalent to finding a smallest path cover in a compacted de Bruijn graph. Using this reduction, we prove a lower bound on the size of the optimal SPSS and propose a greedy method called UST that results in a smaller representation than unitigs and is nearly optimal with respect to our lower bound. We demonstrate the usefulness of the SPSS formulation with two applications of UST. The first one is a compression algorithm, UST-Compress, which we show can store a set of k-mers using an order-of-magnitude less disk space than other lossless compression tools. The second one is an exact static k-mer membership index, UST-FM, which we show improves index size by 10-44% compared to other state-of-the-art low memory indices. Our tool is publicly available at: https://github.com/medvedevgroup/UST/.

show abstract

“…Thus, only reads with the dot product greater than 200 were included in later analysis. We modified squeakr 46 to perform these steps.…”

Section: /11mentioning

confidence: 99%

Long-read Data Revealed Structural Diversity in Human Centromere Sequences

Suzuki

Myers

Morishita

2019

Preprint

View full text Add to dashboard Cite

Centromeres invariably serve as the loci of kinetochore assembly in all eukaryotic cells, but their underlying DNA sequences evolve rapidly. Human centromeres are characterized by their extremely repetitive structures, i.e., higher-order repeats, rendering the region one of the most difficult parts of the genome to assess. Consequently, our understanding of centromere sequence variations across human populations is limited. Here, we analyzed chromosomes 11, 17, and X using long sequencing reads of two European and two Asian genomes, and our results show that human centromere sequences exhibit substantial structural diversity, harboring many novel variant higher-order repeats specific to individuals, while frequent single-nucleotide variants are largely conserved. Our findings add another dimension to our knowledge of centromeres, challenging the notion of stable human centromeres. The discovery of such diversity prompts further deep sequencing of human populations to understand the true nature of sequence evolution in human centromeres. MainCentromeres have been one of the most mysterious parts of the human genome since they were characterized, in the 1970s, as large tracts of 171 bp strings called alpha-satellite monomers 1, 2 . With a growing body of evidence suggesting their relevance to human diseases as sources of genomic instability or as repositories of haplotypes containing causative mutations 3-8 , it has become more important to investigate the underlying sequence variations in centromeres 9, 10 .Human centromeres have nested repeat structures. Namely, a series of distinctively divergent alpha-satellite monomers compose a larger unit called higher-order repeat (HOR) unit, and copies of an HOR unit are tandemly arranged thousands of times to form large, homogeneous HOR arrays. While HOR units are chromosome-specific and consist of two to 34 alphasatellite monomers, copies of an HOR unit are almost identical (95 ∼ 100%) within a chromosome (Figure 1a) [11][12][13][14][15][16][17] . The total HOR array length in each chromosome is known to differ dramatically among individuals 7, 18 or across human populations 19,20 . In addition to array length, other types of variation are known to exist within an HOR array, such as structurally variant HORs that consist of different numbers and/or types of alpha-satellite monomers 20-23 and also single-nucleotide variations (SNVs) between the HOR units 20, 24, 25 . One of the major driving forces leading to such centromeric variation has been thought to be structural alterations such as unequal crossovers and/or gene conversions 26,27 .Previous studies have investigated centromeric sequence variations via traditional approaches such as restriction enzymes sensitive to alpha-satellite monomers, Southern blotting, or analysis of k-mers unique to centromeres in short reads obtained in the 1000 Genomes Project 28 , but their observations have remained indirect and were confined to specific types of variations due to technological limitations.Recently, the advent of long-rea...

show abstract

Squeakr: an exact and approximate k-mer counting system

Abstract: Supplementary data are available at Bioinformatics online.

Cited by 70 publications

References 28 publications

Efficient exact associative structure for sequencing data

Efficient exact associative structure for sequencing data

Representation of k-mer sets using spectrum-preserving string sets

Long-read Data Revealed Structural Diversity in Human Centromere Sequences

Contact Info

Product

Resources

About