The application of Hadoop in structural bioinformatics

Alnasir, Jamie J.; Shanahan, H. P.

doi:10.1093/bib/bby106

Cited by 6 publications

(1 citation statement)

References 83 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…YARN is responsible for resource management of the cluster and job scheduling/monitoring. Altogether, HDFS and YARN are able to provide the fault tolerance and data locality of Hadoop clusters (Taylor, 2010 ; Alnasir and Shanahan, 2018 ).…”

Section: Design and Implementationmentioning

confidence: 99%

BigFiRSt: A Software Program Using Big Data Technique for Mining Simple Sequence Repeats From Large-Scale Sequencing Data

Chen

Wang

et al. 2022

Front. Big Data

View full text Add to dashboard Cite

BackgroundSimple Sequence Repeats (SSRs) are short tandem repeats of nucleotide sequences. It has been shown that SSRs are associated with human diseases and are of medical relevance. Accordingly, a variety of computational methods have been proposed to mine SSRs from genomes. Conventional methods rely on a high-quality complete genome to identify SSRs. However, the sequenced genome often misses several highly repetitive regions. Moreover, many non-model species have no entire genomes. With the recent advances of next-generation sequencing (NGS) techniques, large-scale sequence reads for any species can be rapidly generated using NGS. In this context, a number of methods have been proposed to identify thousands of SSR loci within large amounts of reads for non-model species. While the most commonly used NGS platforms (e.g., Illumina platform) on the market generally provide short paired-end reads, merging overlapping paired-end reads has become a common way prior to the identification of SSR loci. This has posed a big data analysis challenge for traditional stand-alone tools to merge short read pairs and identify SSRs from large-scale data.ResultsIn this study, we present a new Hadoop-based software program, termed BigFiRSt, to address this problem using cutting-edge big data technology. BigFiRSt consists of two major modules, BigFLASH and BigPERF, implemented based on two state-of-the-art stand-alone tools, FLASH and PERF, respectively. BigFLASH and BigPERF address the problem of merging short read pairs and mining SSRs in the big data manner, respectively. Comprehensive benchmarking experiments show that BigFiRSt can dramatically reduce the execution times of fast read pairs merging and SSRs mining from very large-scale DNA sequence data.ConclusionsThe excellent performance of BigFiRSt mainly resorts to the Big Data Hadoop technology to merge read pairs and mine SSRs in parallel and distributed computing on clusters. We anticipate BigFiRSt will be a valuable tool in the coming biological Big Data era.

show abstract

Section: Design and Implementationmentioning

confidence: 99%

BigFiRSt: A Software Program Using Big Data Technique for Mining Simple Sequence Repeats From Large-Scale Sequencing Data

Chen

Wang

et al. 2022

Front. Big Data

View full text Add to dashboard Cite

show abstract

Applications of alignment-free sequence descriptors in the characterization of sequences in the age of big data: a case study with Zika virus, SARS, MERS, and COVID-19

Sen

Dey

Vračko

et al. 2023

Big Data Analytics in Chemoinformatics and Bioinformatics

View full text Add to dashboard Cite

iProX in 2021: connecting proteomics data sharing with big data

Chen

Liu

et al. 2021

Nucleic Acids Research

310

169

View full text Add to dashboard Cite

The rapid development of proteomics studies has resulted in large volumes of experimental data. The emergence of big data platform provides the opportunity to handle these large amounts of data. The integrated proteome resource, iProX (https://www.iprox.cn), which was initiated in 2017, has been greatly improved with an up-to-date big data platform implemented in 2021. Here, we describe the main iProX developments since its first publication in Nucleic Acids Research in 2019. First, a hyper-converged architecture with high scalability supports the submission process. A hadoop cluster can store large amounts of proteomics datasets, and a distributed, RESTful-styled Elastic Search engine can query millions of records within one second. Also, several new features, including the Universal Spectrum Identifier (USI) mechanism proposed by ProteomeXchange, RESTful Web Service API, and a high-efficiency reanalysis pipeline, have been added to iProX for better open data sharing. By the end of August 2021, 1526 datasets had been submitted to iProX, reaching a total data volume of 92.42TB. With the implementation of the big data platform, iProX can support PB-level data storage, hundreds of billions of spectra records, and second-level latency service capabilities that meet the requirements of the fast growing field of proteomics.

show abstract

The application of Hadoop in structural bioinformatics

Cited by 6 publications

References 83 publications

BigFiRSt: A Software Program Using Big Data Technique for Mining Simple Sequence Repeats From Large-Scale Sequencing Data

BigFiRSt: A Software Program Using Big Data Technique for Mining Simple Sequence Repeats From Large-Scale Sequencing Data

Applications of alignment-free sequence descriptors in the characterization of sequences in the age of big data: a case study with Zika virus, SARS, MERS, and COVID-19

iProX in 2021: connecting proteomics data sharing with big data

Contact Info

Product

Resources

About