Data-Intensive Computing Infrastructure Systems for Unmodified Biological Data Analysis Pipelines

Abstract

Abstract. Biological data analysis is typically implemented using a deep pipeline that combines a wide array of tools and databases. These pipelines must scale to very large datasets, and consequently require parallel and distributed computing. It is therefore important to choose a hardware platform and underlying data management and processing systems well suited for processing large datasets. There are many infrastructure systems for such data-intensive computing. However, in our experience, most biological data analysis pipelines do not leverage these systems.We give an overview of data-intensive computing infrastructure systems, and describe how we have leveraged these for: (i) scalable fault-tolerant computing for large-scale biological data; (ii) incremental updates to reduce the resource usage required to update large-scale compendium; and (iii) interactive data analysis and exploration. We provide lessons learned and describe problems we have encountered during development and deployment. We also provide a literature survey on the use of data-intensive computing systems for biological data processing. Our results show how unmodified biological data analysis tools can benefit from infrastructure systems for data-intensive computing.

Large-scale biological meta-database management

“…The main advantage is improved performance and scalability for I/O bound jobs. The main disadvantage is that applications may need to be modified to utilize such a platform fully [30,25,26].…”

“…Systems such as Troilkatt [26,30] are designed to execute workflows on clusters built for data-intensive computing [31]. Compared to HPC clusters these have storage distributed on the compute nodes, and data processing systems that utilize such distributed storage.…”

“…We provide a background describing biological data analysis implementation, configuration, and execution. Further examples can be found in [25], [26].…”

Integrating Data-Intensive Computing Systems with Biological Data Analysis Frameworks

“…We discuss the limitations and lessons learned utilizing data-intensive systems for biological data processing in [18].…”