A History-Based Resource Manager for Genome Analysis Workflows Applications on Clusters with Heterogeneous Nodes

Abstract

Bioinformatics workflows require large amounts of resources and are commonly executed in clusters. Determining the adequate amount of resources for bioinformatics applications is a tricky matter, since the resource usage of a single application might vary substantially from one execution to the next. Resource management systems in clusters don't consider these variations and subsequent needs. As a result, the computing power offered by clusters is not harnessed properly, compromising both application performance and resource efficiency. To tackle these issues, we propose a History-Based Resource Manager for bioinformatics workflows applications running on clusters with heterogeneous nodes. The proposed resource manager features a prediction model that generates multiple performance predictions for each job under different combinations of cluster resources. Furthermore, the proposed resource manager includes a scheduling algorithm that considers the degree of multiprogramming of the nodes, scheduling combinations of applications for simultaneous same-node execution upon their compatibility. To test the proposed resource manager, we process two workloads formed by different amounts of workflows made up by common bioinformatics applications. Results prove that for the given cases, the proposed resource manager improves the performance obtained with SLURM, using First Come First Served policy. The proposal shows an average workflow makespan improvement range between 28 and 35%, averaging 32%, an average workflow efficiency improvement range between 75 and 83%, averaging 79%, and an average resource usage improvement range between 96 and 101%, averaging 99%. Furthermore, the pro-Funded by the Spanish Economy ministry.

Parallel Programming in Bioinformatics: Some Interesting Approaches

“…The second paper, "A History-based Resource Manager for Genome Analysis Workflows Applications on Clusters with Heterogeneous Nodes" by F. Badosa, A. Espinosa, C. Acevedo, G. Vera, and A. Ripoll (see [3]), is focused on the bioinformatics workflows and how to manage the computational resources required by them when they are executed. Bioinformatics workflows require large amounts of resources and are commonly executed in clusters.…”