Scientific Workflows

Abstract

Modern scientific collaborations have opened up the opportunity to solve complex problems that require both multidisciplinary expertise and large-scale computational experiments. These experiments typically consist of a sequence of processing steps that need to be executed on selected computing platforms. Execution poses a challenge, however, due to (1) the complexity and diversity of applications , (2) the diversity of analysis goals , (3) the heterogeneity of computing platforms , and (4) the volume and distribution of data . A common strategy to make these in silico experiments more manageable is to model them as workflows and to use a workflow management system to organize their execution. This article looks at the overall challenge posed by a new order of scientific experiments and the systems they need to be run on, and examines how this challenge can be addressed by workflows and workflow management systems. It proposes a taxonomy of workflow management system (WMS) characteristics, including aspects previously overlooked. This frames a review of prevalent WMSs used by the scientific community, elucidates their evolution to handle the challenges arising with the emergence of the “fourth paradigm,” and identifies research needed to maintain progress in this area.

Towards a scientific data framework to support scientific model development

“…The framework investigated in this paper shares some features with scientific workflow management systems (WMSs). Workflows are "a set of interrelated computational and data-handling tasks designed to achieve a specific goal" [32] and a WMS aids in the automation of those operations by managing their execution and information exchange. Scientific workflows are typically data-intensive workflows and scientific WMSs have been proven crucial in data-driven science [5].…”

“…Unlike WMSs, we do not directly address the development and management of new experiments ("in silico" experimentation) or models, under the assumption that these activities occur externally to the framework. Workflows can become complex and resource intensive, therefore WMSs face problems such as scheduling and resource allocation in distributed environments [32]. A workflow consists of several computational tasks linked by data dependencies and their management by WMSs involves challenges such as resource provisioning, provenance management, performance variation, parallelism and fault tolerance [19,46].…”

A characterization of workflow management systems for extreme-scale applications

“…Computational and data analysis pipelines normally consist of sequential and concurrent tasks that process streams of data [11]. Tasks with data dependencies may run simultaneously operating on different data sets, following the concurrent execution model.…”

“…To address this problem, several recent surveys [11,12,13,14,15,16,17,18] have been compiled to help users compare and contrast different WMSs based on certain key properties and capabilities of WMSs. These surveys focused mostly on the characterization of the following properties: support for conditional structures (e.g., if and switch statements, while loops, etc.)…”

Interoperable and scalable data analysis with microservices: Applications in Metabolomics

“…With increasing data size and number of samples, the analysis process becomes intractable for desktop computers due to requirements on compute cores, memory, storage etc. As a result, large-scale computing infrastructures have become important components in scientific projects (8) .…”