Data-centric iteration in dynamic workflows

Abstract

SummaryDynamic workflows are scientific workflows to support computational science simulations, typically using dynamic processes based on runtime scientific data analyses. They require the ability of adapting the workflow, at runtime, based on user input and dynamic steering. Supporting data-centric iteration is an important step towards dynamic workflows because user interaction with workflows is iterative. However, current support for iteration in scientific workflows is static and does not allow for changing data at runtime. In this paper, we propose a solution based on algebraic operators and a dynamic execution model to enable workflow adaptation based on user input and dynamic steering. We introduce the concept of iteration lineage that makes provenance data management consistent with dynamic iterative workflow changes. Lineage enables scientists to interact with workflow data and configuration at runtime through an API that triggers steering. We evaluate our approach using a novel and real large-scale workflow for uncertainty quantification on a 640-core cluster. The results show impressive execution time savings from 2.5 to 24 days, compared to noniterative workflow execution. We verify that the maximum overhead introduced by our iterative model is less than 5% of execution time. Also, our proposed steering algorithms are very efficient and run in less than 1 millisecond, in the worst-case scenario.

Raw data queries during data-intensive parallel workflow execution

“…Then, another external program would have to be used or programmed to browse cav1.xmf to find the element cav.1.008.h5 inside the file and continue from there, all manually. In addition, changing a loop condition (e.g., TMAX) requires an external loop execution control, to be able to dynamically change it, and react to these changes, like keeping consistency in dataflow generation, as we show in [15].…”

“…In particular, the workflow algebra proposed by Ogasawara et al [13] is capable of modeling entities as datasets and its relationships. The execution control is based on data dependency synchronization, but the algebra also has loop control and branch control operations [15]. For instance, if a data transformation T i generates I i+1 that is a subset of I i , we may say that this T i behaves like a Filter operator.…”

“…We used Chiron SWMS [12] and its data-centric algebra [13] to query provenance data related to domain data using a specialization of W3C PROV [14]. The work in [11] evolved into defining workflow algebraic operations to control and adapt loop conditions [15]. In a study on uncertainty quantification we anticipated a convergent state and dynamically changed loop control [15].…”

“…The work in [11] evolved into defining workflow algebraic operations to control and adapt loop conditions [15]. In a study on uncertainty quantification we anticipated a convergent state and dynamically changed loop control [15]. Such actions contributed to reduce execution time in several hours.…”

OpenAlea

“…Connecting an X to an actor transforms a workflow into a lambda function, and allows to express higher-order programming providing control flow behavior using a set of algebraic operators. The three iteration types can be expressed as [7,5]: (1) counting loops without dependencies (map operator), (2) counting loops with dependencies (reduce and for operators) and (3) conditional loops (while operator). In Figure 2(a), the dataflow variables and the algebraic operators are represented using yellow and white nodes, respectively.…”