Fine-Grain Interoperability of Scientific Workflows in Distributed Computing Infrastructures

Abstract

Today there exist a wide variety of scientific workflow management systems, each designed to fulfill the needs of a certain scientific community. Unfortunately, once a workflow application has been designed in one particular system it becomes very hard to share it with users working with different systems. Portability of workflows and interoperability between current systems barely exists. In this work, we present the fine-grained interoperability solution proposed in the SHIWA European project that brings together four representative European workflow systems: ASKALON, MOTEUR, WS-PGRADE, and Triana. The proposed interoperability is realised at two levels of abstraction: abstract and concrete. At the abstract level, we propose a generic Interoperable Workflow Intermediate Representation (IWIR) that can be used as a common bridge for translating workflows between different languages independent of the underlying distributed computing infrastructure. At the concrete level, we propose a bundling technique that aggregates the abstract IWIR representation and concrete task representations to enable workflow instantiation, execution and scheduling. We illustrate case studies using two real-workflow applications designed in a native environment and then translated and executed by a foreign workflow system in a foreign distributed computing infrastructure.

Infrastructure Aware Scientific Workflows and Infrastructure Aware Workflow Managers in Science Gateways

“…From a practical point of view (simplifying [8]), approaches for workflow interoperability can be divided into ones centered around workflow languages and graph based ones. Language based interoperability solutions tackle with translating from one workflow description language to another or provide an intermediate common language [18]. This is a fine grained solution and the "white box" approach in this paper corresponds to this option.…”

A Formal Approach to Support Interoperability in Scientific Meta-workflows

“…Such experiments commonly involve analyzing large volumes of data and can be run on a variety of computing platforms including high performance computing infrastructures such as clusters, grids and clouds. Scientific workflows are often composed of control and data flow statements and rules which perform the analytics required to achieve the intended experimentation [1]. A typical scientific workflow is composed of one or more individual tasks each of which requires certain number of inputs and generates the respective output(s) after performing its intended function.…”

A novel approach to user-steering in scientific workflows

“…One workflow system is able to invoke another workflow engine through the use of the SHIWA interface, and the Shiwa Portal facilitates the publishing and sharing of reusable workflows [7]. The fine-grained approach [8] deals with language interoperability by defining and Interoperable Workflow Intermediate Representation (IWIR) [9] language for translating workflows (ASKALON, P-Grade, MOTEUR and Triana) from one DCI to another, thus creating a cross-compiler for workflows.…”