CALQ: compression of quality values of aligned sequencing data

Voges, Jan; Östermann, Jörn; Hernáez, Mikel

doi:10.1093/bioinformatics/btx737

Cited by 20 publications

(25 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When it comes to aligned data, an MPEG-G encoder could use a compression method comparable to that of DeeZ [8], which is able to compress a 437 GB H. Sapiens SAM file to about 63 GB, as compared to 75 GB by CRAM (Scramble) or 106 GB by BAM [8]. Regarding quantization of quality values, methods like QVZ [9] and CALQ [10] could be applied yielding overall compression gains of 10x over BAM, while preserving, or even improving, variant calling performance [11].…”

Section: Compression Capabilitiesmentioning

confidence: 99%

“…In the case of unaligned reads, an MPEG-G compliant encoder is free to choose any beneficial quantization scheme. This includes quantization schemes of recently published research such as [9,10,11,12,13]. The specific quantization scheme used is signaled to a decoder by the means of a Quality Value Codebook.…”

Section: Compression Modes For Quality Valuesmentioning

confidence: 99%

“…In the case of aligned reads, MPEG-G introduces an additional dimension to fine-tune quality value quantization: codebooks can be chosen per genomic position, i.e., per locus [13]. Therefore, one Quality Value Codebook Identifier per genomic position is sent to a decoder along with the Quality Value Indexes.…”

Section: Compression Modes For Quality Valuesmentioning

confidence: 99%

See 2 more Smart Citations

An introduction to MPEG-G, the new ISO standard for genomic information representation

Albert

Paridaens²,

Voges³

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

The MPEG-G standardization initiative is a coordinated international effort to specify a compressed data format that enables large scale genomic data to be processed, transported and shared. The standard consists of a set of specifications (i.e., a book) describing: i) a normative format syntax, and ii) a normative decoding process to retrieve the information coded in a compliant file or bitstream. Such decoding process enables the use of leading-edge compression technologies that have exhibited significant compression gains over currently used formats for storage of unaligned and aligned sequencing reads. Additionally, the standard provides a wealth of much needed functionality, such as selective access, data aggregation, application programming interfaces to the compressed data, standard interfaces to support data protection mechanisms, support for streaming and a procedure to assess the conformance of implementations. ISO/IEC is engaged in supporting the maintenance and availability of the standard specification, which guarantees the perenniality of applications using MPEG-G. Finally, the standard ensures interoperability and integration with existing genomic information processing pipelines by providing support for conversion from the FASTQ/SAM/BAM file formats.In this paper we provide an overview of the MPEG-G specification, with particular focus on the main advantages and novel functionality it offers. As the standard only specifies the decoding process, encoding performance, both in terms of speed and compression ratio, can vary depending on specific encoder implementations, and will likely improve during the lifetime of MPEG-G. Hence, the performance statistics provided here are only indicative baseline examples of the technologies included in the standard.

show abstract

Section: Compression Capabilitiesmentioning

confidence: 99%

Section: Compression Modes For Quality Valuesmentioning

confidence: 99%

See 1 more Smart Citation

An introduction to MPEG-G, the new ISO standard for genomic information representation

Albert

Paridaens²,

Voges³

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…This idea has been further improved by the more recent QVZ and QVZ 2 compressors [14,10]. Besides the binning and statistical inference approaches, there are other efforts which exploit the information contained in the readout nucleotide sequences [11,23,21]. For example, the Quartz compressor [23] sets the quality scores of the most frequent k-mers to a predefined high value with the motivation that if a specific nucleotide sequence is observed many times, then its correctness does not need any further verification from the quality scores.…”

Section: Previous Studiesmentioning

confidence: 99%

A Two-Level Scheme for Quality Score Compression

Voges

Fotouhi

Ostermann

et al. 2018

Journal of Computational Biology

Self Cite

View full text Add to dashboard Cite

Previous studies on quality score compression can be classified into two main lines: lossy schemes and lossless schemes. Lossy schemes enable a better management of computational resources. Thus, in practice, and for preliminary analyses, bioinformaticians may prefer to work with a lossy quality score representation. However, the original quality scores might be required for a deeper analysis of the data. Hence, it might be necessary to keep them; in addition to lossy compression this requires lossless compression as well. We developed a space-efficient hierarchical representation of quality scores, QScomp, which allows the users to work with lossy quality scores in routine analysis, without sacrificing the capability of reaching the original quality scores when further investigations are required. Each quality score is represented by a tuple through a novel decomposition. The first and second dimensions of these tuples are separately compressed such that the first-level compression is a lossy scheme. The compressed information of the second dimension allows the users to extract the original quality scores. Experiments on real data reveal that the downstream analysis with the lossy part-spending only 0.49 bits per quality score on average-shows a competitive performance, and that the total space usage with the inclusion of the compressed second dimension is comparable to the performance of competing lossless schemes.

show abstract

“…"Vertical" compression takes a slice through an aligned dataset in the SAM format (Li et al, 2009) to determine which qualities to keep and which to discard, as used in CALQ (Voges et al, 2017), or via hashing techniques on unaligned data in Leon (Benoit et al, 2015) and GeneCodeq (Greenfield et al, 2016). Traditional loss measures, such as mean squared error, will appear very high, but these tools focus on minimising the changes in post-processed data (variant calling).…”

Section: Introductionmentioning

confidence: 99%

Crumble: reference free lossy compression of sequence quality values

Bonfield

McCarthy

Durbin

2018

Preprint

View full text Add to dashboard Cite

Motivation: The bulk of space taken up by NGS sequencing CRAM files consists of per-base quality values. Most of these are unnecessary for variant calling, offering an opportunity for space saving. Results: On the CHM1+CHM13 test set, a 17 fold reduction in quality storage can be achieved while maintaining variant calling accuracy. Availability: Crumble is OpenSource and can be obtained from https://github.com/jkbonfield/crumble.

show abstract

CALQ: compression of quality values of aligned sequencing data

Abstract: Supplementary data are available at Bioinformatics online.

Cited by 20 publications

References 30 publications

An introduction to MPEG-G, the new ISO standard for genomic information representation

An introduction to MPEG-G, the new ISO standard for genomic information representation

A Two-Level Scheme for Quality Score Compression

Crumble: reference free lossy compression of sequence quality values

Contact Info

Product

Resources

About