Representation of $$k$$-mer Sets Using Spectrum-Preserving String Sets

Rahman, Amatur; Medvedev, Paul

doi:10.1007/978-3-030-45257-5_10

Cited by 24 publications

(54 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A disjoint-node path cover for G(S) is a set of paths where each node belongs to exactly one path, e.g., a set of unitigs, maximal unitigs, or maximal stitched unitigs [Rahman and Medvedev, 2020] (also known as simplitigs [Břinda et al, 2021]). We denote such a cover with S .…”

Section: Preliminariesmentioning

confidence: 99%

“…The paths can be represented using an FM-index [Ferragina and Manzini, 2000] for very compact space [Chikhi et al, 2014, Rahman andMedvedev, 2020]. The practical efficiency of the FM-index mainly depends on how many samples of the suffix-array are kept in the index.…”

Section: Related Workmentioning

confidence: 99%

“…Since k should be chosen large enough to allow good k-mer specificity, 2(k − 1) will be much larger than log 2 (N ) in practice, even for the largest genomes. For example, we use k = 31 in our experiments, as done in many other works [Almodaresi et al, 2018, Marchet et al, 2021, Rahman and Medvedev, 2020, Bingmann et al, 2019, whereas log 2 (N ) is around 30 − 33 for collections with billions of kmers (see also Table 2 at page 6). The use of absolute offsets can almost halve the space overhead for the indexing of super-k-mers in such cases.…”

Section: Sparse Hashingmentioning

confidence: 99%

“…For the scope of this work, it is sufficient to point out that: (1) many algorithms have been proposed to build de Bruijn graphs efficiently [Chikhi et al, 2016, Khan and Patro, 2021, Khan et al, 2021 from which these paths can be extracted for indexing purposes; (2) not surprisingly, essentially all state-of-the-art dictionaries for k-mers -that we briefly review in Section 3 -are based on the principle of indexing such collections of paths [Chikhi et al, 2014, Almodaresi et al, 2018, Rahman and Medvedev, 2020, Marchet et al, 2021. We also follow this direction.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Sparse and Skew Hashing of K-Mers

Pibiri

2022

Preprint

View full text Add to dashboard Cite

Motivation: A dictionary of k-mers is a data structure that stores a set of n distinct k-mers and supports membership queries. This data structure is at the hearth of many important tasks in computational biology. High-throughput sequencing of DNA can produce very large k-mer sets, in the size of billions of strings - in such cases, the memory consumption and query efficiency of the data structure is a concrete challenge. Results: To tackle this problem, we describe a compressed and associative dictionary for k-mers, that is: a data structure where strings are represented in compact form and each of them is associated to a unique integer identifier in the range [0,n). We show that some statistical properties of k-mer minimizers can be exploited by minimal perfect hashing to substantially improve the space/time trade-off of the dictionary compared to the best-known solutions. Availability: The C++ implementation of the dictionary is available at https://github.com/jermp/sshash.

show abstract

Section: Preliminariesmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Sparse Hashingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Sparse and Skew Hashing of K-Mers

Pibiri

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…A way to tackle this problem is to only store counters, discarding k-mers information. This idea is supported by the fact that in many applications, queried k-mers come from partially assembled reads or succinct representations, such as SPSS [7,8], or Colored de Bruijn graphs [9], that is, only k-mers present in the original data are queried for their frequencies.…”

Section: Introductionmentioning

confidence: 99%

Set-Min sketch: a probabilistic map for power-law distributions with application to k-mer annotation

Shibuya

Belazzougui

Kucherov

2020

Preprint

View full text Add to dashboard Cite

Motivation: In many bioinformatics pipelines, k-mer counting is often a required step, with existing methods focusing on optimizing time or memory usage. These methods usually produce very large count tables explicitly representing k-mers themselves. Solutions avoiding explicit representation of k-mers include Minimal Perfect Hash Functions (MPHFs) or Count-Min sketches. The former is only applicable to static maps not subject to updates, while the latter suffers from potentially very large point-query errors, making it unsuitable when counters are required to be highly accurate. Results: We introduce Set-Min sketch, a sketching technique inspired by Count-Min sketch, for representing associative maps, more specifically, k-mer count tables. We show that Set-Min sketch provides a very low error rate, both in terms of the probability and the size of errors, much lower than a Count-Min sketch of similar dimensions. On the other hand, Set-Min sketches are shown to take up to an order of magnitude less space than MPHF-based solutions, especially for large values of k. Space-efficiency of Set-min takes advantage of the power-law distribution of k-mer counts in genomic datasets.

show abstract

Spectrum Preserving Tilings Enable Sparse and Modular Reference Indexing

Fan

Khan

Pibiri

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The reference indexing problem for $$k$$-mers is to pre-process a collection of reference genomic sequences $$\mathcal {R}$$ so that the position of all occurrences of any queried $$k$$-mer can be rapidly identified. An efficient and scalable solution to this problem is fundamental for many tasks in bioinformatics.In this work, we introduce the spectrum preserving tiling (SPT), a general representation of $$\mathcal {R}$$ that specifies how a set of tiles repeatedly occur to spell out the constituent reference sequences in $$\mathcal {R}$$. By encoding the order and positions where tiles occur, SPTs enable the implementation and analysis of a general class of modular indexes. An index over an SPT decomposes the reference indexing problem for $$k$$-mers into: (1) a $$k$$-mer-to-tile mapping; and (2) a tile-to-occurrence mapping. Recently introduced work to construct and compactly index $$k$$-mer sets can be used to efficiently implement the $$k$$-mer-to-tile mapping. However, implementing the tile-to-occurrence mapping remains prohibitively costly in terms of space. As reference collections become large, the space requirements of the tile-to-occurrence mapping dominates that of the $$k$$-mer-to-tile mapping since the former depends on the amount of total sequence while the latter depends on the number of unique $$k$$-mers in $$\mathcal {R}$$.To address this, we introduce a class of sampling schemes for SPTs that trade off speed to reduce the size of the tile-to-reference mapping. We implement a practical index with these sampling schemes in the tool . When indexing over 30,000 bacterial genomes, reduces the size of the tile-to-occurrence mapping from 86.3 GB to 34.6 GB while incurring only a 3.6$$\times $$ slowdown when querying $$k$$-mers from a sequenced readset.Availability: is implemented in Rust and available at https://github.com/COMBINE-lab/pufferfish2.

show abstract

Representation of $$k$$-mer Sets Using Spectrum-Preserving String Sets

Cited by 24 publications

References 38 publications

Sparse and Skew Hashing of K-Mers

Sparse and Skew Hashing of K-Mers

Set-Min sketch: a probabilistic map for power-law distributions with application to k-mer annotation

Spectrum Preserving Tilings Enable Sparse and Modular Reference Indexing

Contact Info

Product

Resources

About