An effective extension of the applicability of alignment-free biological sequence comparison algorithms with Hadoop

Cattaneo, Giuseppe; Petrillo, Umberto Ferraro; Giancarlo, Raffaele; Roscigno, Gianluca

doi:10.1007/s11227-016-1835-3

Cited by 25 publications

(15 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To further speed-up ‘Multi-SpaM’, we have parallelized our software to run on multiple cores; in Table 1 , we report both wall-clock and ‘CPU’ run times. It should be straight-forward to adapt our software to run on distributed systems, as has been done for other alignment-free approaches ( 66 , 67 ).…”

Section: Discussionmentioning

confidence: 99%

‘Multi-SpaM’: a maximum-likelihood approach to phylogeny reconstruction using multiple spaced-word matches and quartet trees

Dencker

Leimeister

Gerth

et al. 2019

NAR Genomics and Bioinformatics

View full text Add to dashboard Cite

Word-based or ‘alignment-free’ methods for phylogeny inference have become popular in recent years. These methods are much faster than traditional, alignment-based approaches, but they are generally less accurate. Most alignment-free methods calculate ‘pairwise’ distances between nucleic-acid or protein sequences; these distance values can then be used as input for tree-reconstruction programs such as neighbor-joining. In this paper, we propose the first word-based phylogeny approach that is based on ‘multiple’ sequence comparison and ‘maximum likelihood’. Our algorithm first samples small, gap-free alignments involving four taxa each. For each of these alignments, it then calculates a quartet tree and, finally, the program ‘Quartet MaxCut’ is used to infer a super tree for the full set of input taxa from the calculated quartet trees. Experimental results show that trees produced with our approach are of high quality.

show abstract

Section: Discussionmentioning

confidence: 99%

‘Multi-SpaM’: a maximum-likelihood approach to phylogeny reconstruction using multiple spaced-word matches and quartet trees

Dencker

Leimeister

Gerth

et al. 2019

NAR Genomics and Bioinformatics

View full text Add to dashboard Cite

show abstract

“…Normalization of the background and including inexact matches increases the time complexity. Alignment-free methods based on k -mers lend themselves to parallel algorithms, and parallel computational methods have been applied to achieve speedup and scalability for alignment-free methods (90). When k is large, memory is a main limitation for storing k -mer counts and computing alignment-free measures (91).…”

Section: Applications Of the Alignment-free Methods To Comparative Gementioning

confidence: 99%

Alignment-Free Sequence Analysis and Applications

Ren

Bai

et al. 2018

Annu. Rev. Biomed. Data Sci.

View full text Add to dashboard Cite

Genome and metagenome comparisons based on large amounts of next generation sequencing (NGS) data pose significant challenges for alignment-based approaches due to the huge data size and the relatively short length of the reads. Alignment-free approaches based on the counts of word patterns in NGS data do not depend on the complete genome and are generally computationally efficient. Thus, they contribute significantly to genome and metagenome comparison. Recently, novel statistical approaches have been developed for the comparison of both long and shotgun sequences. These approaches have been applied to many problems including the comparison of gene regulatory regions, genome sequences, metagenomes, binning contigs in metagenomic data, identification of virus-host interactions, and detection of horizontal gene transfers. We provide an updated review of these applications and other related developments of word-count based approaches for alignment-free sequence analysis.

show abstract

“…The modern high-throughput technologies produce high amounts of sequence collections of data, and several methodologies have been proposed for their efficient storage and analysis [34,35]. Recently, approaches based on MapReduce and big data technologies have been proposed (see, e.g., [36], and [3] for a complete review on this topic). An important issue in this context is the computation of k-mer statistics, that becomes challenging when sets of sequences at a genomic scale are involved.…”

Section: Big Data Based Approaches For the Analysis Of Biological Seqmentioning

confidence: 99%

Analyzing big datasets of genomic sequences: fast and scalable collection of k-mer statistics

et al. 2019

Self Cite

View full text Add to dashboard Cite

Background Distributed approaches based on the MapReduce programming paradigm have started to be proposed in the Bioinformatics domain, due to the large amount of data produced by the next-generation sequencing techniques. However, the use of MapReduce and related Big Data technologies and frameworks (e.g., Apache Hadoop and Spark) does not necessarily produce satisfactory results, in terms of both efficiency and effectiveness. We discuss how the development of distributed and Big Data management technologies has affected the analysis of large datasets of biological sequences. Moreover, we show how the choice of different parameter configurations and the careful engineering of the software with respect to the specific framework under consideration may be crucial in order to achieve good performance, especially on very large amounts of data. We choose k -mers counting as a case study for our analysis, and Spark as the framework to implement FastKmer, a novel approach for the extraction of k -mer statistics from large collection of biological sequences, with arbitrary values of k . Results One of the most relevant contributions of FastKmer is the introduction of a module for balancing the statistics aggregation workload over the nodes of a computing cluster, in order to overcome data skew while allowing for a full exploitation of the underlying distributed architecture. We also present the results of a comparative experimental analysis showing that our approach is currently the fastest among the ones based on Big Data technologies, while exhibiting a very good scalability. Conclusions We provide evidence that the usage of technologies such as Hadoop or Spark for the analysis of big datasets of biological sequences is productive only if the architectural details and the peculiar aspects of the considered framework are carefully taken into account for the algorithm design and implementation.

show abstract

An effective extension of the applicability of alignment-free biological sequence comparison algorithms with Hadoop

Cited by 25 publications

References 41 publications

‘Multi-SpaM’: a maximum-likelihood approach to phylogeny reconstruction using multiple spaced-word matches and quartet trees

‘Multi-SpaM’: a maximum-likelihood approach to phylogeny reconstruction using multiple spaced-word matches and quartet trees

Alignment-Free Sequence Analysis and Applications

Analyzing big datasets of genomic sequences: fast and scalable collection of k-mer statistics

Contact Info

Product

Resources

About