Efficient Execution of Scientific Workflows in the Cloud Through Adaptive Caching

Abstract

Many scientific experiments are now carried on using scientific workflows, which are becoming more and more data-intensive and complex. We consider the efficient execution of such workflows in the cloud. Since it is common for workflow users to reuse other workflows or data generated by other workflows, a promising approach for efficient workflow execution is to cache intermediate data and exploit it to avoid task re-execution. In this paper, we propose an adaptive caching solution for data-intensive workflows in the cloud. Our solution is based on a new scientific workflow management architecture that automatically manages the storage and reuse of intermediate data and adapts to the variations in task execution times and output data size. We evaluated our solution by implementing it in the OpenAlea system and performing extensive experiments on real data with a data-intensive application in plant phenotyping. The results show that adaptive caching can yield major performance gains, e.g., up to a factor of 3.5 with 6 workflow re-executions.

Caching for dataset-based workloads with heterogeneous file sizes

“…For example, in [16], the authors simply rely on XCache [19] and its internal implementation of the LRU policy. To the best of our knowledge, only paper [20] explores the usage of a caching technique significantly different from LRU for scientific workloads. The authors propose an adaptive caching solution that is only suitable for tasks with high re-execution rates, which are not present in the WLCG.…”

“…In the function cache_file, the second argument indicates the fraction of the file that needs to be cached. The final step is to add a fraction of the first rectangle that did not fit (lines [18][19][20]. Note how we are implicitly solving the fractional knapsack problem, and then we either match or exceed the optimal solution of the original knapsack problem [27, Sec.…”

Cache-aware scheduling of scientific workflows in a multisite cloud

“…Workflow simplification is performed by the workflow manager before execution, transforming the workflow into an executable workflow and considering the metadata, input, and cache data location. It is based on the workflow simplification method presented in [23].…”