Compressive biological sequence analysis and archival in the era of high-throughput sequencing technologies

Giancarlo, Raffaele; Rombo, Simona E.; Utro, Filippo

doi:10.1093/bib/bbt088

Cited by 52 publications

(34 citation statements)

References 85 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The first algorithms from 2009 were soon followed by more mature proposals, which will be presented below, focusing on their indexing capabilities. More information on genome data compressors and indexes can be found in the recent surveys (Vyverman et al, 2012;Deorowicz and Grabowski, 2013;Giancarlo et al, 2013). Mäkinen et al (2010) added index functionalities to compressed DNA sequences: display (which can also be called the random access functionality) returning the substring specified by its start and end position, count telling the number of times the given pattern occurs in the text, and locate listing the positions of the pattern in the text.…”

Section: Introductionmentioning

confidence: 99%

Indexes of Large Genome Collections on a PC

2014

View full text Add to dashboard Cite

The availability of thousands of individual genomes of one species should boost rapid progress in personalized medicine or understanding of the interaction between genotype and phenotype, to name a few applications. A key operation useful in such analyses is aligning sequencing reads against a collection of genomes, which is costly with the use of existing algorithms due to their large memory requirements. We present MuGI, Multiple Genome Index, which reports all occurrences of a given pattern, in exact and approximate matching model, against a collection of thousand(s) genomes. Its unique feature is the small index size, which is customisable. It fits in a standard computer with 16–32 GB, or even 8 GB, of RAM, for the 1000GP collection of 1092 diploid human genomes. The solution is also fast. For example, the exact matching queries (of average length 150 bp) are handled in average time of 39 µs and with up to 3 mismatches in 373 µs on the test PC with the index size of 13.4 GB. For a smaller index, occupying 7.4 GB in memory, the respective times grow to 76 µs and 917 µs. Software is available at http://sun.aei.polsl.pl/mugi under a free license. Data S1 is available at PLOS One online.

show abstract

Section: Introductionmentioning

confidence: 99%

Indexes of Large Genome Collections on a PC

2014

View full text Add to dashboard Cite

show abstract

“…The modern high-throughput technologies produce high amounts of sequence collections of data, and several methodologies have been proposed for their efficient storage and analysis [34,35]. Recently, approaches based on MapReduce and big data technologies have been proposed (see, e.g., [36], and [3] for a complete review on this topic).…”

Section: Big Data Based Approaches For the Analysis Of Biological Seqmentioning

confidence: 99%

“…An important issue in this context is the computation of k-mer statistics, that becomes challenging when sets of sequences at a genomic scale are involved. Due to the importance of this task in several applications (e.g., genome assembly [37] and alignment-free sequence analysis [34,35]) many methods that use shared-memory multi processor architectures or distributed computing have been proposed. The basic pattern followed by most of these methods is to maintain a shared data structure (typically, a hash table) to be updated according to the k-mers extracted from a collection of input files by one or more concurrent tasks.…”

Section: Big Data Based Approaches For the Analysis Of Biological Seqmentioning

confidence: 99%

Analyzing big datasets of genomic sequences: fast and scalable collection of k-mer statistics

et al. 2019

View full text Add to dashboard Cite

Background Distributed approaches based on the MapReduce programming paradigm have started to be proposed in the Bioinformatics domain, due to the large amount of data produced by the next-generation sequencing techniques. However, the use of MapReduce and related Big Data technologies and frameworks (e.g., Apache Hadoop and Spark) does not necessarily produce satisfactory results, in terms of both efficiency and effectiveness. We discuss how the development of distributed and Big Data management technologies has affected the analysis of large datasets of biological sequences. Moreover, we show how the choice of different parameter configurations and the careful engineering of the software with respect to the specific framework under consideration may be crucial in order to achieve good performance, especially on very large amounts of data. We choose k -mers counting as a case study for our analysis, and Spark as the framework to implement FastKmer, a novel approach for the extraction of k -mer statistics from large collection of biological sequences, with arbitrary values of k . Results One of the most relevant contributions of FastKmer is the introduction of a module for balancing the statistics aggregation workload over the nodes of a computing cluster, in order to overcome data skew while allowing for a full exploitation of the underlying distributed architecture. We also present the results of a comparative experimental analysis showing that our approach is currently the fastest among the ones based on Big Data technologies, while exhibiting a very good scalability. Conclusions We provide evidence that the usage of technologies such as Hadoop or Spark for the analysis of big datasets of biological sequences is productive only if the architectural details and the peculiar aspects of the considered framework are carefully taken into account for the algorithm design and implementation.

show abstract

“…In these file formats, other information, such as identifiers and quality scores, are added to the raw genomic or protein sequences. It is worth noting that there are other formats for storing biological data, such as Illumina Export format, but they were developed for targeting a specific technology [4].…”

Section: File Formatsmentioning

confidence: 99%

“…Also, the costs associated with the storage, processing and transmission of HTS data are higher compared to sequence generation, that makes the situation more complicated [1]. Compression is a solution that is able to overcome these challenges, by reducing the storage size and processing costs, such as I/O bandwidth, as well as increasing transmission speed [2][3][4].…”

Section: Introductionmentioning

confidence: 99%

A Survey on Data Compression Methods for Biological Sequences

2016

View full text Add to dashboard Cite

Abstract:The ever increasing growth of the production of high-throughput sequencing data poses a serious challenge to the storage, processing and transmission of these data. As frequently stated, it is a data deluge. Compression is essential to address this challenge-it reduces storage space and processing costs, along with speeding up data transmission. In this paper, we provide a comprehensive survey of existing compression approaches, that are specialized for biological data, including protein and DNA sequences. Also, we devote an important part of the paper to the approaches proposed for the compression of different file formats, such as FASTA, as well as FASTQ and SAM/BAM, which contain quality scores and metadata, in addition to the biological sequences. Then, we present a comparison of the performance of several methods, in terms of compression ratio, memory usage and compression/decompression time. Finally, we present some suggestions for future research on biological data compression.

show abstract

Compressive biological sequence analysis and archival in the era of high-throughput sequencing technologies

Cited by 52 publications

References 85 publications

Indexes of Large Genome Collections on a PC

Indexes of Large Genome Collections on a PC

Analyzing big datasets of genomic sequences: fast and scalable collection of k-mer statistics

A Survey on Data Compression Methods for Biological Sequences

Contact Info

Product

Resources

About