Adaptable, metadata rich IO methods for portable high performance IO

Lofstead, Jay; Zheng, Fang; Klasky, Scott; Schwan, Karsten

doi:10.1109/ipdps.2009.5161052

Cited by 126 publications

(77 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They primarily focus on fast and asynchronous data movement off simulation nodes to lessen the impact of expensive I/O operations. Examples of this approach include ADIOS [49], DataSpaces [50], and Glean [51].…”

Section: In Situ Workflowsmentioning

confidence: 99%

A characterization of workflow management systems for extreme-scale applications

Silva

Filgueira

Pietri

et al. 2017

Future Generation Computer Systems

116

View full text Add to dashboard Cite

Automation of the execution of computational tasks is at the heart of improving scientific productivity. Over the last years, scientific workflows have been established as an important abstraction that captures data processing and computation of large and complex scientific applications. By allowing scientists to model and express entire data processing steps and their dependencies, workflow management systems relieve scientists from the details of an application and manage its execution on a computational infrastructure. As the resource requirements of today's computational and data science applications that process vast amounts of data keep increasing, there is a compelling case for a new generation of advances in high-performance computing, commonly termed as extreme-scale computing, which will bring forth multiple challenges for the design of workflow applications and management systems. This paper presents a novel characterization of workflow management systems using features commonly associated with extreme-scale computing applications. We classify 15 popular workflow management systems in terms of workflow execution models, heterogeneous computing environments, and data access methods. The paper also surveys workflow applications and identifies gaps for future research on the road to extreme-scale workflows and management systems.

show abstract

Section: In Situ Workflowsmentioning

confidence: 99%

A characterization of workflow management systems for extreme-scale applications

Silva

Filgueira

Pietri

et al. 2017

Future Generation Computer Systems

116

View full text Add to dashboard Cite

show abstract

“…To improve the I/O performance of climate models, previous efforts and I/O libraries include a message passing interface-input/output (MPI-IO, Corbetty et al, 1996), the Network Common Data Form (netCDF, Rew and Davis, 1990), parallel netCDF (PnetCDF, Li et al, 2003), parallel I/O (PIO, Dennis et al, 2012), and adaptable I/O system (ADIOS, Lofstead et al, 2009, Lofstead et al, 2008.…”

Section: Introductionmentioning

confidence: 99%

A fast input/output library for high-resolution climate models

Wang

Yang

et al. 2014

Geosci. Model Dev.

View full text Add to dashboard Cite

Abstract. We describe the design and implementation of climate fast input/output (CFIO), a fast input/output (I/O) library for high-resolution climate models. CFIO provides a simple method for modelers to overlap the I/O phase with the computing phase automatically, so as to shorten the running time of numerical simulations. To minimize the code modifications required for porting, CFIO provides similar interfaces and features to parallel Network Common Data Form (PnetCDF), which is one of the most widely used I/O libraries in climate models. We deployed CFIO in three highresolution climate models, including two ocean models (POP and LICOM) and one sea ice model (CICE). The experimental results show that CFIO improves the performance of climate models significantly versus the original serial I/O approach. When running with CFIO at 0.1 • resolution with about 1000 CPU cores, we managed to reduce the running time by factors of 7.9, 4.6 and 2.0 for POP, CICE, and LI-COM, respectively. We also compared the performance of CFIO against two existing libraries, PnetCDF and parallel I/O (PIO), in different scenarios. For scenarios with both data output and computations, CFIO decreases the I/O overhead compared to PnetCDF and PIO.

show abstract

“…The benefits for using staging have been proven in several papers [16,9,18]. Our group has taken the data staging approach using asynchronous I/O and hosting operations on data prior to writing to persistent storage [1,20].…”

Section: Introductionmentioning

confidence: 99%

“…The last decision can be handled in the IO API rather than requiring any intervention from the scientific application programmer. A system like ADIOS [15] that affords incorporating these custom protocols makes adding this protocol transparent.…”

Section: Introductionmentioning

confidence: 99%

Resilient data staging through MxN distributed transactions.

Schwan

Oldfield²,

Lofstead³

et al. 2011

Self Cite

View full text Add to dashboard Cite

Scientific computing-driven discoveries are frequently driven from workflows that use persistent storage as a staging area for data between operations. With the bad and progressively worse bandwidth vs. data size issues as we continue towards exascale, eliminating persistent storage through techniques like data staging will both enable these workflows to continue online, but also enable more interactive workflows reducing the time to scientific discoveries. Data staging has shown to be an effective way for applications running on high-end computing platforms to offload expensive I/O operations and to manage the tremendous amounts of data they produce. 3This data staging approach, however, lacks the ACID style guarantees traditional straightto-disk methods provide. Distributed transactions are a proven way to add ACID properties to data movements, however distributed transactions follow 1xN data movement semantics, where our highly parallel HPC environments employ MxN data movement semantics. In this paper we present a novel protocol that extends distributed transaction terminology to include MxN semantics which allows our data staging areas to benefit from ACID properties. We show that with our protocol we can provide resilient data staging with a limited performance penalty over current data staging implementations.4

show abstract

Adaptable, metadata rich IO methods for portable high performance IO

Cited by 126 publications

References 11 publications

A characterization of workflow management systems for extreme-scale applications

A characterization of workflow management systems for extreme-scale applications

A fast input/output library for high-resolution climate models

Resilient data staging through MxN distributed transactions.

Contact Info

Product

Resources

About