Performance evaluation and tuning of BioPig for genomic analysis

Shi, Lizhen; Zhong, Wang; Yu, Weikuan; Meng, Xiangru

doi:10.1145/2831244.2831252

Cited by 2 publications

(3 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There're some work about Pig or Hadoop tuning, such as [20] [21] [22] [23]. [20] presented a detailed step-by-step tuning process of K-mer counting on Hadoop.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Combining Hadoop with MPI to Solve Metagenomics Problems that are both Data- and Compute-intensive

Lin

Meng

et al. 2017

Int J Parallel Prog

Self Cite

View full text Add to dashboard Cite

Abstract. Metagenomics, the study of all microbial species cohabitants in an environment, often produces large amount of sequence data varying from several GBs to a few TBs. Analysing metagenomics data involving several steps, some steps are data intensive, and some are compute intensive. Typical bioinformatics pipelines attempt to analyse the entire data set on computer servers with several terabytes of RAM, which is very inefficient. To overcome this limit, here we propose a MapReduce based solution to partition the data based on their species of origin. We implemented the solution using BioPig, an analytic toolkit for large-scale genomic sequence data based on Apache Hadoop and Pig. We simplified data types and logic design, compressed k-mer storage and combined Hadoop with MPI to improve the computational performance. After these optimizations, we achieved up to 193x speedup for the rate-limiting step and 8x speedup for the entire pipeline, respectively. The optimized software is also capable to process datasets that are 16 times larger on the same hardware platform. Results from this case study suggest the combined Hadoop with MPI approach has great potential in large genomics applications that are both data-intensive and compute-intensive.

show abstract

“…There're some work about Pig or Hadoop tuning, such as [20] [21] [22] [23]. [20] presented a detailed step-by-step tuning process of K-mer counting on Hadoop.…”

Section: Discussionmentioning

confidence: 99%

“…[20] presented a detailed step-by-step tuning process of K-mer counting on Hadoop. [21] and [22] tuned Hadoop performance at different software stack levels as well as hardware.…”

Section: Discussionmentioning

confidence: 99%

Combining Hadoop with MPI to Solve Metagenomics Problems that are both Data- and Compute-intensive

Lin

Meng

et al. 2017

Int J Parallel Prog

Self Cite

View full text Add to dashboard Cite

show abstract

“…Incorrect reading in huge input can be eliminated by using Brujin graphs while performing sequence assembling. This can be used to align multiple sequences, as the size of k-mer increases most of the computation time will be for a read operation (Shi et al, 2015). Hence k-mer counting is a complex problem in genomic data analysis.…”

Section: Introductionmentioning

confidence: 99%

Hybrid Approach for Enhancing Performance of Genomic Data for Stream Matching

Gururaj

Siddesh

2021

International Journal of Cognitive Informatics and Natural Intelligence

View full text Add to dashboard Cite

In gene expression analysis, the expression levels of thousands of genes are analyzed, such as separate stages of treatments or diseases. Identifying particular gene sequence pattern is a challenging task with respect to performance issues. The proposed solution addresses the performance issues in genomic stream matching by involving assembly and sequencing. Counting the k-mer based on k-input value and while performing DNA sequencing tasks, the researches need to concentrate on sequence matching. The proposed solution addresses performance issue metrics such as processing time for k-mer counting, number of operations for matching similarity, memory utilization while performing similarity search, and processing time for stream matching. By suggesting an improved algorithm, Revised Rabin Karp(RRK) for basic operation and also to achieve more efficiency, the proposed solution suggests a novel framework based on Hadoop MapReduce blended with Pig & Apache Tez. The measure of memory utilization and processing time proposed model proves its efficiency when compared to existing approaches.

show abstract

Performance evaluation and tuning of BioPig for genomic analysis

Cited by 2 publications

References 8 publications

Combining Hadoop with MPI to Solve Metagenomics Problems that are both Data- and Compute-intensive

Combining Hadoop with MPI to Solve Metagenomics Problems that are both Data- and Compute-intensive

Hybrid Approach for Enhancing Performance of Genomic Data for Stream Matching

Contact Info

Product

Resources

About