A Simple Statistical Algorithm for Biological Sequence Compression

Cao, Minh Duc; Dix, Trevor I.; Allison, Lloyd; Mears, Christopher

doi:10.1109/dcc.2007.7

Cited by 56 publications

(19 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Based on partial matches of subsets of the input, this model predicts the next symbols in the sequence. High compression rates are possible if the model always indicates high probabilities for the next symbol, i.e., if the prediction is reliable [12,13].…”

Section: Basic Techniquesmentioning

confidence: 99%

Trends in Genome Compression

Wandelt

Bux

Leser

2014

CBIO

View full text Add to dashboard Cite

Technological advancements in high-throughput sequencing have lead to a tremendous increase in the amount of genomic data produced. With the cost being down to 2,000 USD for a single human genome, sequencing dozens of individuals is a task that is feasible even for smaller project or organizations already today. However, generating the sequence is only one issue; another one is storing, managing, and analyzing it. These tasks become more and more challenging due to the sheer size of the data sets and are increasingly considered to be the most severe bottlenecks in larger genome projects. One possible countermeasure is to compress the data; compression reduces costs in terms of requiring less hard disk storage and in terms of requiring less bandwidth if data is shipped to large compute clusters for parallel analysis. Accordingly, sequence compression has recently attracted much interest in the scientific community. In this paper, we explain the different basic techniques for sequence compression, point to distinctions between different compression tasks (e.g., genome versus read compression), and present a comparison of current approaches and tools. To further stimulate progress in genome compression research, we also identify key challenges for future systems.Keywords: genome compression, read compression, survey Key messages of the article:• Overview of trends in genome compression: bit manipulation, dictionarybased, statistical, and referential compression schemes• Wide ranges of compression performance (compression ratio, compression time, memory requirements)• Comparing compression schemes for evaluation purposes is difficult• Many minor improvements in recent contributions

show abstract

Section: Basic Techniquesmentioning

confidence: 99%

Trends in Genome Compression

Wandelt

Bux

Leser

2014

CBIO

View full text Add to dashboard Cite

show abstract

“…In 1993 the first specialized DNA compressor was proposed (Grumbach and Tahi, 1993). Since then, numerous DNA compressors were developed (e.g., Cao et al, 2007, Li et al, 2013, Benoit et al, 2015, Al-Okaily et al, 2017. In our experience only two compressors pass the practicality threshold: DELIMINATE (Mohammed et al, 2012) and MFCompress (Pinho and Pratas, 2014).…”

Section: Introductionmentioning

confidence: 87%

Nucleotide Archival Format (NAF) enables efficient lossless reference-free compression of DNA sequences

Kryukov

Ueda

Nakagawa

et al. 2018

Preprint

View full text Add to dashboard Cite

DNA sequence databases use compression such as gzip to reduce the required storage space and network transmission time. We describe Nucleotide Archival Format (NAF)a new file format for lossless reference-free compression of FASTA and FASTQ-formatted nucleotide sequences. NAF compression ratio is comparable to the best DNA compressors, while providing dramatically faster decompression. We compared our format with DNA compressors: DELIMINATE and MFCompress, and with general purpose compressors: gzip, bzip2, xz, brotli, and zstd.Availability: NAF compressor and decompressor, as well as format specification are available at https://github.com/KirillKryukov/naf. Format specification is in public domain. Compressor and decompressor are open source under the zlib/libpng license, free for nearly any use.

show abstract

“…Lempel-Ziv-based compression. Statistical algorithms [13,20] derive a predictive model from (a subset of) the input, based on partial matches. If the model always indicates high probabilities for the next symbol, then high compression rates are possible.…”

Section: Related Workmentioning

confidence: 99%

Adaptive efficient compression of genomes

Wandelt

Leser

2012

Algorithms Mol Biol

View full text Add to dashboard Cite

Modern high-throughput sequencing technologies are able to generate DNA sequences at an ever increasing rate. In parallel to the decreasing experimental time and cost necessary to produce DNA sequences, computational requirements for analysis and storage of the sequences are steeply increasing. Compression is a key technology to deal with this challenge. Recently, referential compression schemes, storing only the differences between a to-be-compressed input and a known reference sequence, gained a lot of interest in this field. However, memory requirements of the current algorithms are high and run times often are slow. In this paper, we propose an adaptive, parallel and highly efficient referential sequence compression method which allows fine-tuning of the trade-off between required memory and compression speed. When using 12 MB of memory, our method is for human genomes on-par with the best previous algorithms in terms of compression ratio (400:1) and compression speed. In contrast, it compresses a complete human genome in just 11 seconds when provided with 9 GB of main memory, which is almost three times faster than the best competitor while using less main memory.

show abstract

A Simple Statistical Algorithm for Biological Sequence Compression

Cited by 56 publications

References 24 publications

Trends in Genome Compression

Trends in Genome Compression

Nucleotide Archival Format (NAF) enables efficient lossless reference-free compression of DNA sequences

Adaptive efficient compression of genomes

Contact Info

Product

Resources

About