Compression of Large genomic datasets using COMRAD on Parallel Computing Platform

Abstract

The big data storage is a challenge in a post genome era. Hence, there is a need for high performance computing solutions for managing large genomic data. Therefore, it is of interest to describe a parallel-computing approach using message-passing library for distributing the different compression stages in clusters. The genomic compression helps to reduce the on disk“foot print” of large data volumes of sequences. This supports the computational infrastructure for a more efficient archiving. The approach was shown to find utility in 21 Eukaryotic genomes using stratified sampling in this report. The method achieves an average of 6-fold disk space reduction with three times better compression time than COMRAD.AvailabilityThe source codes are written in C using message passing libraries and are available at https:// sourceforge.net/ projects/ comradmpi/files / COMRADMPI/

Application of signal processing for DNA sequence compression

“…However, not all the methods in Section 5 can provide effective compression for sequences which only share partial similarity in certain portions. Methods that can be applied for characterising partial similarity include COMRAD [52,53], ERGC [54], CoGI [55], MSC [56] and referential compression algorithm [81].…”

“…These rules are then entropy encoded. Example rule-based compression methods are DNA Sequitur [31] and COMRAD [52,53]. In parsing methods, the DNA sequence is usually divided into a number of phrases using a greedy strategy where the phrases denote repeats in the sequence.…”

“…Compression using Redundancy of DNA (COMRAD) [52,53] https:// sourceforge.net/projects/ comradmpi/files/COMRADMPI/ A dictionary that stores repeats of all sequences in the population dataset is constructed iteratively. The repeats are encoded into short words to achieve compression.…”

“…Besides, as clustering is performed on the population sequences, the compression time is significantly increased. Various methods to improve the computational complexity can be investigated, such as the use of parallel computing platform [53].…”

“…In other words, similar subsequences are encoded once only which act as references to their occurrence at other locations of the same sequence or other sequences. Examples of methods in this group include DNAzip [12], RLCSA [46], RLZ [47,48], GRS [49], GReEn [50], iDoComp [51], COMRAD [52,53], ERGC [54], CoGI [55], MSC [56], GDC [57,58], FRESCO [59] and RCC [60]. With the use of appropriate reference sequences, the storage size reduction in some cases can be over 90%.…”