HyperLoom Possibilities for Executing Scientific Workflows on the Cloud

Abstract

We have developed HyperLoom-a platform for defining and executing scientific workflows in large-scale HPC systems. The computational tasks in such workflows often have non-trivial dependency patterns, unknown execution time and unknown sizes of generated outputs. HyperLoom enables to efficiently execute the workflows respecting task requirements and cluster resources agnostically to the shape or size of the workflow. Although HPC infrastructures provide an unbeatable performance, they may be unavailable or too expensive especially for small to medium workloads. Moreover, for some workloads, due to HPCs not very flexible resource allocation policy, the system energy efficiency may not be optimal at some stages of the execution. In contrast, current public cloud providers such as Amazon, Google or Exoscale allow users a comfortable and elastic way of deploying, scaling and disposing a virtualized cluster of almost any size. In this paper, we describe HyperLoom virtualization and evaluate its performance in a virtualized environment using workflows of various shapes and sizes. Finally, we discuss the Hyperloom potential for its expansion to cloud environments.

Hyperparameter search in periodic vehicle routing problem

“…The histogram of solution time is given in Figure 4. The importance of the input parameters was determined as the proportion of the variance of the solution time that can be explained by the given input parameter (using the local regression model LOESS) [13]. This proportion was then recalculated as the relative measure where the importance of the input parameters is relative to the input parameter with the highest degree of importance (100%).…”