SORA: Scalable Overlap-graph Reduction Algorithms for Genome Assembly using Apache Spark in the Cloud

Paul, Alexander J.; Lawrence, Dylan; Song, Myoungkyu; Lim, Seung-Hwan; Pan, Chongle; Ahn, Tae-Hyuk

doi:10.1109/bibm.2018.8621546

Cited by 4 publications

(4 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The experiments are divided into two groups: (a) parallel performance and scalability of diBELLA 2D and (b) performance comparison with related work. In the latter case, we compare the overlap detection of diBELLA 2D with diBELLA 1D [9] and minimap2 [16] written in C for shared memory, while we compare our transitive reduction algorithm with SORA [23] written in Scala on top of Apache Spark [24].…”

Section: Methodsmentioning

confidence: 99%

“…SORA [23] computes transitive reduction of a string graph based on an overlap graph in distributed memory using Apache Spark [24] and the GraphX library [25], which allows parallel computation on distributed graphs in Spark. To the best of our knowledge, SORA is the only other distributed algorithm that computes transitive reduction on overlap graphs, although it was designed for cloud environments.…”

Section: Related Workmentioning

confidence: 99%

“…The speedup of diBELLA 2D over minimap2 is 9.5× × ×, 13.7× × ×, and 20.6× × × at P ={128, 200, 338} for the H. sapiens dataset. Finally, we compare our transitive reduction algorithm with the transitive reduction module of SORA [23], a distributed memory implementation of transitive reduction from overlap graph to string graph based on Apache Spark and GraphX. Our transitive reduction algorithm is currently integrated into our pipeline, so for fairness reasons we do not include startup, shutdown, and I/O time for SORA.…”

Section: B Comparison With the State Of The Artmentioning

confidence: 99%

See 2 more Smart Citations

Parallel String Graph Construction and Transitive Reduction for De Novo Genome Assembly

Guidi¹,

Selvitopi²,

Ellis³

et al. 2020

Preprint

View full text Add to dashboard Cite

One of the most computationally intensive tasks in computational biology is de novo genome assembly, the decoding of the sequence of an unknown genome from redundant and erroneous short sequences. A common assembly paradigm identifies overlapping sequences, simplifies their layout, and creates consensus. Despite many algorithms developed in the literature, the efficient assembly of large genomes is still an open problem.In this work, we introduce new distributed-memory parallel algorithms for overlap detection and layout simplification steps of de novo genome assembly, and implement them in the diBELLA 2D pipeline. Our distributed memory algorithms for both overlap detection and layout simplification are based on linear-algebra operations over semirings using 2D distributed sparse matrices. Our layout step consists of performing a transitive reduction from the overlap graph to a string graph. We provide a detailed communication analysis of the main stages of our new algorithms.diBELLA 2D achieves near linear scaling with over 80% parallel efficiency for the human genome, reducing the runtime for overlap detection by 1.2-1.3× × × for the human genome and 1.5-1.9× × × for C.elegans compared to the state-of-the-art. Our transitive reduction algorithm outperforms an existing distributed-memory implementation by 10.5-13.3× × × for the human genome and 18-29× × × for the C. elegans. Our work paves the way for efficient de novo assembly of large genomes using long reads in distributed memory.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: B Comparison With the State Of The Artmentioning

confidence: 99%

See 1 more Smart Citation

Parallel String Graph Construction and Transitive Reduction for De Novo Genome Assembly

Guidi¹,

Selvitopi²,

Ellis³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…It runs on a spark platform achieving 13X, speeding up for de novo genome assembly faster than the traditional platform. A recent similar study was introduced by [28]. The IBD algorithm addressed pairwise comparison problems that can be used to overlap two reads or align reads against k-mer of the reference genome [29].…”

Section: Related Workmentioning

confidence: 99%

PTHP: Index for Optimizing Genome Assembly Overlapping and Read Alignment

Barakat,

Sallehuddin,

Yuhaniz

et al. 2023

ijmst

View full text Add to dashboard Cite

Unfortunately, sequencing technology can only access the genome sequence as massive numbers of short strings are called reads. The genome assembly process constructs the complete genome from these reads based on the overlapping between the reads, called the de novo approach, or aligns the reads based on their positions in the available reference genome, called the reference-guided approach. Millions of reads search for overlapping or alignment, a well-known data structure problem called all-against-all. Many studies have proposed indexing such as hash index, prefix tree index, and parallelization technique to optimize the overlapping or the read alignment individually. However, due to the massive data amount and the repeats, limitations still affect the index efficiency, requiring more enhancements. This article introduces a new hybrid index named Prefix Tree Hash Partitioned index(PTHP), which combines prefix-tree index, hash index, pigeonhole concept, and parallelization. PTHP index reveals significant results on the simulation and real dataset, reducing the computational time complexity of overlapping and read alignment, thus the assembly time outperforming prefix tree index and hash index. Improving the performance of overlapping and read alignment using the PTHP index reveals great results in optimizing the hybrid genome assembly that combines both.

show abstract

Parallel String Graph Construction and Transitive Reduction for De Novo Genome Assembly

Guidi

Selvitopi

Ellis

et al. 2021

2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

View full text Add to dashboard Cite

SORA: Scalable Overlap-graph Reduction Algorithms for Genome Assembly using Apache Spark in the Cloud

Cited by 4 publications

References 19 publications

Parallel String Graph Construction and Transitive Reduction for De Novo Genome Assembly

Parallel String Graph Construction and Transitive Reduction for De Novo Genome Assembly

PTHP: Index for Optimizing Genome Assembly Overlapping and Read Alignment

Parallel String Graph Construction and Transitive Reduction for De Novo Genome Assembly

Contact Info

Product

Resources

About