SOLiDzipper: A High Speed Encoding Method for the Next-Generation Sequencing Data

Abstract

Background:Next-generation sequencing (NGS) methods pose computational challenges of handling large volumes of data. Although cloud computing offers a potential solution to these challenges, transferring a large data set across the internet is the biggest obstacle, which may be overcome by efficient encoding methods. When encoding is used to facilitate data transfer to the cloud, the time factor is equally as important as the encoding efficiency. Moreover, to take advantage of parallel processing in cloud computing, a parallel technique to decode and split compressed data in the cloud is essential. Hence in this review, we present SOLiDzipper, a new encoding method for NGS data.Methods:The basic strategy of SOLiDzipper is to divide and encode. NGS data files contain both the sequence and non-sequence information whose encoding efficiencies are different. In SOLiDzipper, encoded data are stored in binary data block that does not contain the characteristic information of a specific sequence platform, which means that data can be decoded according to a desired platform even in cases of Illumina, Solexa or Roche 454 data.Results:The main calculation time using Crossbow was 173 minutes when 40 EC2 nodes were involved. In that case, an analysis preparation time of 464 minutes is required to encode data in the latest DNA compression method like G-SQZ and transmit it on a 183 Mbit/s bandwidth. However, it takes 194 minutes to encode and transmit data with SOLiDzipper under the same bandwidth conditions. These results indicate that the entire processing time can be reduced according to the encoding methods used, under the same network bandwidth conditions. Considering the limited network bandwidth, high-speed, high-efficiency encoding methods such as SOLiDzipper can make a significant contribution to higher productivity in labs seeking to take advantage of the cloud as an alternative to local computing.Availability:http://szipper.dinfree.com. Academic/non-profit: Binary available for direct download at no cost. For-profit: Submit request for for-profit license from the web-site.

Compression of FASTQ and SAM Format Sequencing Data

“…There has been considerable work on compression of sequencing data, with some researchers specializing only on sequence compression [6], [7] or quality value compression [8], [9] with others supporting full FASTQ file compression; G-SQZ [10], SlimGene [11], SOLiDzipper [12], DSRC [13], Quip [14], SCALCE [15] and KungFQ [16]. Related to this is work on SAM/BAM [17] compression including Goby (F. Campagne, http://campagnelab.org/software/goby/ accessed on July 19, 2012), CRAM [18], SAMZIP [19] and NGC [20].…”

A Parallel Algorithm for Compression of Big Next-Generation Sequencing Datasets

“…This behavior is typical for parallel algorithms computing very small datasets [17]. In the case of the 1TB dataset, we conclude that our moderate size cluster, although equipped with a high performance parallel file system, is not able to keep a linear speedup and hence a desired efficiency when 8 or more processing units are concurrently accessing non-contiguous regions of a big dataset 8 .…”

“…Among all the datasets 10GB has the maximum throughput of 8,441MB/s (8.4GB/s) when compressed with 128 processes, which correlates with its superlinear speedup. 128 processing units were used to achieve the 8 This conclusion is based on results obtained comparing the reading times with those of reading contiguous regions of the 1TB datset…”

“…Keeping these NGS datasets intact before the processing stage is important due to the preservation of key elements that may become meaningful in future analyses. Storing and transferring these massive amounts of data (generated very fast and cheaply) represents a challenge that has led to creation of highly specialized compression techniques such as Quip [3], G-SQZ [4], DSRC [5], KungFQ [6], SeqDB [7], SOLiDzipper [8]. These algorithms take into account the characteristics of the FASTQ format [9] in which the DNA sequence read and its corresponding per base quality score are structured and stored.…”

A Hybrid MPI-OpenMP Strategy to Speedup the Compression of Big Next-Generation Sequencing Datasets

“…Algorithms such as Quip [2], G-SQZ [3], DSRC [4], KungFQ [5], SeqDB [6], LW-FQZip [7], LFQC [8], LEON [9], SCALCE [10] and SOLiDzipper [11] take into account the nature of DNA sequences and use different data compression techniques to achieve good ratios. But these algorithms process the data sequentially and do not fully utilize the processing powers of the newest computing resources, such as GPUs or CPU's multi-core technologies, to speed up the compression of an increasing amount of genomic data.…”