Compression of genomic sequencing reads via hash-based reordering: algorithm and analysis

Chandak, Shubham; Tatwawadi, Kedar; Weissman, Tsachy

doi:10.1093/bioinformatics/btx639

Cited by 39 publications

(27 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the evaluation we used the state-of-the-art competitors, i.e., FaStore 12 , Spring 14 , and Minicom 15 . We resigned from testing some other good compressors like BEETL 8 , Orcom 10 , AssembleTrie 22 , and HARC 13 as the previous works demonstrated that they perform worse than the picked tools. The older utilities are not competitive in terms of compression ratio, as was demonstrated in the recent papers 12,14 (see also Table 1 for experiments with one of our datasets).…”

Section: Results Tools and Datasetsmentioning

confidence: 99%

See 1 more Smart Citation

FQSqueezer: k-mer-based compression of sequencing data

Deorowicz

2020

Sci Rep

View full text Add to dashboard Cite

the amount of data produced by modern sequencing instruments that needs to be stored is huge. Therefore it is not surprising that a lot of work has been done in the field of specialized data compression of FASTQ files. The existing algorithms are, however, still imperfect and the best tools produce quite large archives. We present FQSqueezer, a novel compression algorithm for sequencing data able to process single-and paired-end reads of variable lengths. It is based on the ideas from the famous prediction by partial matching and dynamic Markov coder algorithms known from the general-purposecompressors world. The compression ratios are often tens of percent better than offered by the state-ofthe-art tools. the drawbacks of the proposed method are large memory and time requirements.

show abstract

Section: Results Tools and Datasetsmentioning

confidence: 99%

“…In 12 , it was shown how to group reads from a bit larger genome regions. Significantly better results were, however, obtained in three recent articles presenting HARC 13 , Spring 14 , and Minicom 15 . The attempts differ in details, but are based on similar ideas.…”

mentioning

confidence: 83%

FQSqueezer: k-mer-based compression of sequencing data

Deorowicz

2020

Sci Rep

View full text Add to dashboard Cite

show abstract

“…This mode is suitable, for example, for long term storage of raw sequencing data. Examples of preprocessing technologies of which the data output can be represented using the decoder syntax defined in MPEG-G for this mode of operation include those presented in ORCOM [13], HARC [14], FaStore [7] and, in general, all (future) preprocessing technologies that cluster reads based on common patterns.…”

Section: Compression Modes For Raw Sequencing Datamentioning

confidence: 99%

An introduction to MPEG-G, the new ISO standard for genomic information representation

Albert

Paridaens²,

Voges³

et al. 2018

Preprint

View full text Add to dashboard Cite

The MPEG-G standardization initiative is a coordinated international effort to specify a compressed data format that enables large scale genomic data to be processed, transported and shared. The standard consists of a set of specifications (i.e., a book) describing: i) a normative format syntax, and ii) a normative decoding process to retrieve the information coded in a compliant file or bitstream. Such decoding process enables the use of leading-edge compression technologies that have exhibited significant compression gains over currently used formats for storage of unaligned and aligned sequencing reads. Additionally, the standard provides a wealth of much needed functionality, such as selective access, data aggregation, application programming interfaces to the compressed data, standard interfaces to support data protection mechanisms, support for streaming and a procedure to assess the conformance of implementations. ISO/IEC is engaged in supporting the maintenance and availability of the standard specification, which guarantees the perenniality of applications using MPEG-G. Finally, the standard ensures interoperability and integration with existing genomic information processing pipelines by providing support for conversion from the FASTQ/SAM/BAM file formats.In this paper we provide an overview of the MPEG-G specification, with particular focus on the main advantages and novel functionality it offers. As the standard only specifies the decoding process, encoding performance, both in terms of speed and compression ratio, can vary depending on specific encoder implementations, and will likely improve during the lifetime of MPEG-G. Hence, the performance statistics provided here are only indicative baseline examples of the technologies included in the standard.

show abstract

“…As a result, computational methods that reduce memory usage, e.g., by representing genomic data more compactly, or making inference on the fly by processing genomic data in an on-line manner are of high demand. Among these approaches, lossless compression methods on raw, mapped or indexed data [2,3,4,5,6,7] have been highly successful; for example, recent standardization efforts by MPEG-G [8,9] or GA4GH [10]), especially in the context of raw (FASTQ) and mapped (SAM/BAM) read collections based on current generation compression methods, have demonstrated that it is possible to reduce the size of genomic data by an order of magnitude. Similarly, on-line genomic data processing methods (e.g.…”

Section: Introductionmentioning

confidence: 99%

Sketching Algorithms for Genomic Data Analysis and Querying in a Secure Enclave

Koçkan

Zhu

Dokmai

et al. 2018

Preprint

View full text Add to dashboard Cite

Current practices in collaborative genomic data analysis (e.g. PCAWG [1]) necessitate all involved parties to exchange individual patient data and perform all analysis locally, or use a trusted server for maintaining all data to perform analysis in a single site (e.g. the Cancer Genome Collaboratory). Since both approaches involve sharing genomic sequence data -which is typically not feasible due to privacy issues, collaborative data analysis remains to be a rarity in genomic medicine.In order to facilitate efficient and effective collaborative or remote genomic computation we introduce SkSES (Sketching algorithms for Secure Enclave based genomic data analysiS), a computational framework for performing data analysis and querying on multiple, individually encrypted genomes from several institutions in an untrusted cloud environment. Unlike other techniques for secure/privacy preserving genomic data analysis, which typically rely on sophisticated cryptographic techniques with prohibitively large computational overheads, SkSES utilizes the secure enclaves supported by current generation microprocessor architectures such as Intel's SGX. The key conceptual contribution of SkSES is its use of sketching data structures that can fit in the limited memory available in a secure enclave.While streaming/sketching algorithms have been developed for many applications in computer science, their feasibility in genomics has remained largely unexplored. On the other hand, even though privacy and security issues are becoming critical in genomic medicine, available cryptographic techniques based on, e.g. homomorphic encryption or garbled circuits, fail to address the performance demands of this rapidly growing field. The alternative offered by Intel's SGX, a combination of hardware and software solutions for secure data analysis, is severely limited by the relatively small size of a secure enclave, a private region of the memory protected from other processes. SkSES addresses this limitation through the use of sketching data structures to support efficient secure and privacy preserving SNP analysis across individually encrypted VCF files from multiple institutions. In particular SkSES provides the users the ability to query for the "k most significant SNPs" among any set of user specified SNPs and any value of k -even when the total number of SNPs to be maintained is far beyond the memory capacity of the secure enclave. Results: We tested SkSES on the complete iDASH-2017 competition data set comprised of 1000 case and 1000 control samples related to an unknown phenotype. SkSES was able to identify the top SNPs with respect to the χ 2 statistic, among any user specified subset of SNPs across this data set of 2000 individually encrypted complete human genomes quickly and accurately -demonstrating the feasibility of secure and privacy preserving computation for genomic medicine via Intel's SGX.

show abstract

Compression of genomic sequencing reads via hash-based reordering: algorithm and analysis

Abstract: Supplementary material are available at Bioinformatics online. The proposed algorithm is available for download at https://github.com/shubhamchandak94/HARC.

Cited by 39 publications

References 17 publications

FQSqueezer: k-mer-based compression of sequencing data

FQSqueezer: k-mer-based compression of sequencing data

An introduction to MPEG-G, the new ISO standard for genomic information representation

Sketching Algorithms for Genomic Data Analysis and Querying in a Secure Enclave

Contact Info

Product

Resources

About