DataSpaces: an interaction and coordination framework for coupled simulation workflows

Abstract

Emerging high-performance distributed computing environments are enabling new end-to-end formulations in science and engineering that involve multiple interacting processes and data-intensive application workflows. For example, current fusion simulation efforts are exploring coupled models and codes that simultaneously simulate separate application processes, such as the core and the edge turbulence. These components run on different high performance computing resources, need to interact at runtime with each other and with services for data monitoring, data analysis and visualization, and data archiving. As a result, they require efficient and scalable support for dynamic and flexible couplings and interactions, which remains a challenge. This paper presents DataSpaces a flexible interaction and coordination substrate that addresses this challenge. DataSpaces essentially implements a semantically specialized virtual shared space abstraction that can be associatively accessed by all components and services in the application workflow. It enables live data to be extracted from running simulation components, indexes this data online, and then allows it to be monitored, queried and accessed by other components and services via the space using semantically meaningful operators. The underlying data transport is asynchronous, low-overhead and largely memory-to-memory. The design, C. Docan ( ) · M. Parashar

Virtual Observation System for Earth System Model: An Application to ACME Land Model Simulations

“…The latter requires further development of data staging nodes within the system. In this extension, Dataspaces library (Docan et al, 2012) could be used to allocate and manage data staging nodes and handle push and pull operations between the VOS components, whereas Fastbit library (Wu, 2005) could be used for data indexing and query processing within these nodes, and CCI can still provide a twoway communication between simulation and analysis components to enable simulation steering.…”

A characterization of workflow management systems for extreme-scale applications

“…They primarily focus on fast and asynchronous data movement off simulation nodes to lessen the impact of expensive I/O operations. Examples of this approach include ADIOS [49], DataSpaces [50], and Glean [51].…”

Visualization and Analysis for Near-Real-Time Decision Making in Distributed Workflows

“…SYSTEM IMPLEMENTATION Our visualization system is spread across two different geographic locations, The A*STAR Computational Resource Centre in Singapore and Georgia Tech in Atlanta, and is a synthesis of a variety of different tools and frameworks. On a high level this system consists of the simulation and data manipulation routines located in Singapore, the interactive visualization and analysis routines located at Georgia Tech, and a connection over the wide area network using the ADIOS [8] middleware and the the ICEE [9], FlexPath [10], and DataSpaces [11] transport methods. Combined, these technologies allowed us to perform memory to memory data delivery from one side of the pipeline to the other.…”