Rethinking data management for big data scientific workflows

Abstract

Scientific workflows consist of tasks that operate on input data to generate new data products that are used by subsequent tasks. Workflow management systems typically stage data to computational sites before invoking the necessary computations. In some cases data may be accessed using remote I/O. There are limitations with these approaches, however. First, the storage at a computational site may be limited and not able to accommodate the necessary input and intermediate data. Second, even if there is enough storage, it is sometimes managed by a filesystem with limited scalability. In recent years, object stores have been shown to provide a scalable way to store and access large datasets, however, they provide a limited set of operations (retrieve, store and delete) that do not always match the requirements of the workflow tasks. In this paper, we show how scientific workflows can take advantage of the capabilities of object stores without requiring users to modify their workflowbased applications or scientific codes. We present two general approaches, one that exclusively uses object stores to store all the files accessed and generated by a workflow, while the other relies on the shared filesystem for caching intermediate data sets. We have implemented both of these approaches in the Pegasus Workflow Management System and have used them to execute workflows in variety of execution environments ranging from traditional supercomputing environments that have a shared filesystem to dynamic environments like Amazon AWS and the Open Science Grid that only offer remote object stores. As a result, Pegasus users can easily migrate their applications from a shared filesystem deployment to one using object stores without changing their application codes.

Analyzing related raw data files through dataflows

“…However, it does not support raw data file access. Similarly, current solutions for provenance, such as Pegasus Lite [11] and Wings [22], do not address dataflow queries as we show in this paper. Chiron implements a datacentric workflow algebra.…”

Pegasus, a workflow management system for science automation

“…The PegasusLite [46] workflow engine was developed for the case in which jobs are executed in a non-shared filesystem en-vironment. In such deployments, the worker nodes on a cluster do not share a file system between themselves or between them and the data staging server.…”

Dynamic workflow support in gUSE

“…[17] A task level challenge for workflow management systems is to develop a flexible data management solution that allows for late binding of data. Tasks can discover input data at runtime, and possibly choose to stage the data from one of many locations.…”