Semantics-based distributed I/O for mpiBLAST

Abstract

BLAST is a widely used software toolkit for genomic sequence search. mpiBLAST is a freely available, open-source parallelization of BLAST that uses database segmentation to allow different worker processes to search (in parallel) unique segments of the database. After searching, the workers write their output to a filesystem. While mpiBLAST has been shown to achieve high performance in clusters with fast local filesystems, its I/O processing remains a concern for scalability, especially in systems having limited I/O capabilities such as distributed filesystems spread across a wide-area network. Thus, we present ParaMEDIC-a novel environment that uses applicationspecific semantic information to compress I/O data and improve performance in distributed environments. Specifically, for mpiBLAST, ParaMEDIC partitions worker processes into compute and I/O workers. Compute workers, instead of directly writing the output to the filesystem, the workers process the output using semantic knowledge about the application to generate metadata and write the metadata to the filesystem. I/O workers, which physically reside closer to the actual storage, then process this metadata to re-create the actual output and write it to the filesystem. This approach allows ParaMEDIC to reduce I/O time, thus accelerating mpiBLAST by as much as 25-fold.

OrthoInspector: comprehensive orthology analysis and visual exploration

“…The most time-consuming step in all orthology prediction algorithms is the generation of the Blast all-versus-all searches for each new update. In spite of the efforts aimed at developing faster parallelized Blast methods [ 40 , 41 ], the Blast all-versus-all computational requirements grow quadratically with the addition of new proteomes. Therefore, one of our future goals will be to develop an incremental update process, minimizing the number of distance calculations required between the thousands of sequences present in the previous version of the database.…”

Data parallelism in bioinformatics workflows using Hydra

“…However, CloudBLAST approach is disconnected from the concept of scientific workflows and it is not concerned about capturing and analyzing provenance data. G-BLAST [30] framework implements a solution based on grid services for supporting the submission of mpiBLAST [31] jobs to cluster systems. In spite of dealing with BLAST parallel execution, G-BLAST adopts a different fragmentation schema from that used in our work: it chooses to fragment the database rather than input file.…”

Global‐scale distributed I/O with ParaMEDIC

“…In our previous work [7,8], we provided a detailed description of ParaMEDIC. Here we present a brief summary of that work.…”

“…To resolve such issues on a global scale, we proposed a new, non‐traditional approach for distributed I/O known as ParaMEDIC (Parallel Metadata Environment for Distributed I/O and Computing) 6–8. ParaMEDIC uses application‐specific semantic information to process the data generated by treating it as a collection of high‐level abstract objects, rather than as a generic byte‐stream.…”