kmcEx: memory-frugal and retrieval-efficient encoding of counted k-mers

Abstract

Motivation K-mers along with their frequency have served as an elementary building block for error correction, repeat detection, multiple sequence alignment, genome assembly, etc., attracting intensive studies in k-mer counting. However, the output of k-mer counters itself is large; very often, it is too large to fit into main memory, leading to highly narrowed usability. Results We introduce a novel idea of encoding k-mers as well as their frequency, achieving good memory saving and retrieval efficiency. Specifically, we propose a Bloom filter-like data structure to encode counted k-mers by coupled-bit arrays—one for k-mer representation and the other for frequency encoding. Experiments on five real datasets show that the average memory-saving ratio on all 31-mers is as high as 13.81 as compared with raw input, with 7 hash functions. At the same time, the retrieval time complexity is well controlled (effectively constant), and the false-positive rate is decreased by two orders of magnitude. Availability and implementation The source codes of our algorithm are available at github.com/lzhLab/kmcEx. Supplementary information Supplementary data are available at Bioinformatics online.

Efficient Mining of Variants From Trios for Ventricular Septal Defect Association Study

“…For instance, the 31-mers having a count larger than one of the HapMap sample NA12878 (()) take 90-Gb space on disk. To solve this problem, we have designed a novel coupled Bloom Filter-based algorithm achieving high memory saving ratio and good retrieval efficiency (Jiang et al, 2019). Let f max be the maximum frequency in K , which can be represented by at most h bits (in binary).…”

“…Based on the above steps, K f , K m , and K c can be saved into B f , B m , and B c economically; more details are shown in Jiang et al (2019).…”

Enabling Genomics Pipelines in Commodity Personal Computers With Flash Storage

“…Counting the frequencies of k-mers is an algorithm that is widely used in many areas of genomics (Xiao et al, 2018 ); from genome assembly and error detection to sequence alignment and variant calling (Kelley et al, 2010 ; Li et al, 2010 ). Others (Marçais and Kingsford, 2011 ; Rizk et al, 2013 ; Audano and Vannberg, 2014 ; Deorowicz et al, 2015 ; Li and Yan, 2015 ; Jiang et al, 2019 ) have explored ways to optimize k-mer counting with reduced memory and storage. While these k-mer counting algorithms process a single sample, SMUFIN processes k-mer counters of normal and tumoral samples of the same patient together, potentially making the memory footprint even bigger.…”