High end scientific codes with computational I/O pipelines

Zheng, Fang; Cao, Jianting; Dayal, Jai; Eisenhauer, Greg; Schwan, Karsten; Wolf, Matthew; Abbasi, Hasan; Klasky, Scott; Podhorszki, Norbert

doi:10.1145/2110205.2110210

Cited by 4 publications

(2 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An important benefit of this approach is that once applications utilize ADIOS for I/O, in situ and in transit workflows can be instantiated without requiring any further modifications to the application code. Transports such as DataSpaces [DPK12, DZJ*14], FlexPath [DCE*13, ZCD*11] and IceE [CWW*13] all follow the same principle ‐ data is buffered in memory at the application node, additional in situ processing can be applied to this buffer, and the processed data is moved to auxiliary nodes for an in transit workflow.…”

Section: In Depth Analysis Of Four In Situ Infrastructuresmentioning

confidence: 99%

In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms

Bauer

Abbasi

Ahrens

et al. 2016

Computer Graphics Forum

144

View full text Add to dashboard Cite

The considerable interest in the high performance computing (HPC) community regarding analyzing and visualization data without first writing to disk, i.e., in situ processing, is due to several factors. First is an I/O cost savings, where data is analyzed/visualized while being generated, without first storing to a filesystem. Second is the potential for increased accuracy, where fine temporal sampling of transient analysis might expose some complex behavior missed in coarse temporal sampling. Third is the ability to use all available resources, CPU's and accelerators, in the computation of analysis products. This STAR paper brings together researchers, developers and practitioners using in situ methods in extreme-scale HPC with the goal to present existing methods, infrastructures, and a range of computational science and engineering applications using in situ analysis and visualization.

show abstract

Section: In Depth Analysis Of Four In Situ Infrastructuresmentioning

confidence: 99%

In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms

Bauer

Abbasi

Ahrens

et al. 2016

Computer Graphics Forum

144

View full text Add to dashboard Cite

show abstract

“…Recent research has explored in-situ, in-memory techniques, based on data staging approaches, as a means of addressing the coordination and data management challenges for complex workflows [22,27,12]. For example, our previous work on DataSpaces [9] is an in-memory data staging implementation and has been used to support in-situ/in-transit workflows [26,25].…”

Section: Introductionmentioning

confidence: 99%

Persistent Data Staging Services for Data Intensive In-situ Scientific Workflows

Romanus

Zhang

Jin

et al. 2016

Proceedings of the ACM International Workshop on Data-Intensive Distributed Computing

Self Cite

View full text Add to dashboard Cite

Scientific simulation workflows executing on very large scale computing systems are essential modalities for scientific investigation. The increasing scales and resolution of these simulations provide new opportunities for accurately modeling complex natural and engineered phenomena. However, the increasing complexity necessitates managing, transporting, and processing unprecedented amounts of data, and as a result, researchers are increasingly exploring data-staging and in-situ workflows to reduce data movement and datarelated overheads. However, as these workflows become more dynamic in their structures and behaviors, data staging and in-situ solutions must evolve to support new requirements. In this paper, we explore how the service-oriented concept can be applied to extreme-scale in-situ workflows. Specifically, we explore persistent data staging as a service and present the design and implementation of DataSpaces as a Service, a service-oriented data staging framework. We use a dynamically coupled fusion simulation workflow to illustrate the capabilities of this framework and evaluate its performance and scalability.

show abstract