Pipel: exploiting resource reorganisation to optimise performance of pipeline-structured applications in the cloud

Abstract

Workflow has become a standard for many scientific applications that are characterized by a collection of processing elements. Particularly, a pipeline application is a type of workflow that receives a set of tasks, which must pass through all processing elements in a linear fashion. However, the strategy of using a fixed number of resources can cause under-or over-provisioning situations, besides not fitting irregular demands. In this context, our idea is to deploy the pipeline application in the cloud, so executing it with a feature that differentiates cloud from other distributed systems: resource elasticity. Thus, we propose Pipel: a reactive elasticity model that uses lower and upper load thresholds and the CPU metric to on-the-fly select the most appropriated number of compute nodes for each stage along the pipeline execution. The results were promising, presenting an average gain of 38% in the application time when comparing non-elastic and elastic executions.