An efficient parallel approach for identifying protein families in large-scale metagenomic data sets

Abstract

Abstract-Metagenomics is the study of environmental microbial communities using state-of-the-art genomic tools. Recent advancements in high-throughput technologies have enabled the accumulation of large volumes of metagenomic data that was until a couple of years back was deemed impractical for generation. A primary bottleneck, however, is in the lack of scalable algorithms and open source software for largescale data processing. In this paper, we present the design and implementation of a novel parallel approach to identify protein families from large-scale metagenomic data. Given a set of peptide sequences we reduce the problem to one of detecting arbitrarily-sized dense subgraphs from bipartite graphs. Our approach efficiently parallelizes this task on a distributed memory machine through a combination of divide-and-conquer and combinatorial pattern matching heuristic techniques. We present performance and quality results of extensively testing our implementation on 160K randomly sampled sequences from the CAMERA environmental sequence database using 512 nodes of a BlueGene/L supercomputer.

Coordinating Computation and I/O in Massively Parallel Sequence Search

“…The experimental results on multiple platforms demonstrate that our integrated scheduling approach allows large-scale sequence search to efficiently scale on general parallel computers. In the future, we plan to generalize our study to other scientific applications with irregular computation and I/O patterns such as parallel HMMER [53] and large-scale protein family identification [54].…”

An OpenMP algorithm and implementation for clustering biological graphs

“…For instance: it can be used to reduce redundancy within sequence repositories; identify complexes within metabolic networks [2]; identify core groups of proteins that constitute a protein family [26,28,29] and in the process also help assign family memberships for newly found peptide candidates [29]; help in the construction of mass spectral libraries for peptides [17]; and can be used to condense the space of plausible computer-generated phylogenetic trees [23].…”

“…In our earlier work, we implemented a serial version of this heuristic, and applied it in the context of metagenomic protein family detection [28]. Put briefly, this approach [28], called pClust, transforms the problem into one of bipartite graph clustering so that the approach developed by Gibson et al originally for web community detection can be used.…”

“…In our earlier work, we implemented a serial version of this heuristic, and applied it in the context of metagenomic protein family detection [28]. Put briefly, this approach [28], called pClust, transforms the problem into one of bipartite graph clustering so that the approach developed by Gibson et al originally for web community detection can be used. The results [28,27] on input sets of size up to 1.2 million amino acid sequences showed both run-time and quality (sensitivity) advantage over approaches that use other heuristics.…”

“…Put briefly, this approach [28], called pClust, transforms the problem into one of bipartite graph clustering so that the approach developed by Gibson et al originally for web community detection can be used. The results [28,27] on input sets of size up to 1.2 million amino acid sequences showed both run-time and quality (sensitivity) advantage over approaches that use other heuristics. Despite these advantages, the implementation of the clustering step (i.e., pClust) is serial and does not scale beyond a graph containing 15K-20K vertices on a desktop computer with 2 GB RAM due to memory requirement.…”