Experiences Applying Data Staging Technology in Unconventional Ways

Lofstead, Jay; Oldfield, Ron; Kordenbrock, Todd

doi:10.1109/ccgrid.2013.27

Cited by 5 publications

(3 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recent work has looked at general-purpose frameworks for developing "data services". Nessie [19,28], DataStager [1], GLEAN [36], and I/O delegation [24] are examples of these frameworks. Although the initial motivation for these frameworks was to "stage" data to avoid bursty I/O patterns [26], it has also shown value as a tool for offloading analysis from a running simulation.…”

Section: Background and Related Workmentioning

confidence: 98%

Evaluation of methods to integrate analysis into a large-scale shock shock physics code

Oldfield

Moreland

Fabian

et al. 2014

Proceedings of the 28th ACM International Conference on Supercomputing

View full text Add to dashboard Cite

Exascale supercomputing will embody many revolutionary changes in the hardware and software of high-performance computing. For example, projected limitations in power and I/O-system performance will fundamentally change visualization and analysis workflows. A traditional post-processing workflow involves storing simulation results to disk and later retrieving them for visualization and data analysis; however, at Exascale, post-processing approaches will not be able to capture the volume or granularity of data necessary for analysis of these extreme-scale simulations. As an alternative, researchers are exploring ways to integrate analysis and simulation without using the storage system. In situ and in transit are two options, but there has not been an adequate evaluation of these approaches to identify strengths, weaknesses, and trade-offs at large scale. This paper provides a detailed performance and scaling analysis of a largescale shock physics code using traditional post-processsing, in situ, and in transit analysis to detect material fragments from a simulated explosion.

show abstract

Section: Background and Related Workmentioning

confidence: 98%

Evaluation of methods to integrate analysis into a large-scale shock shock physics code

Oldfield

Moreland

Fabian

et al. 2014

Proceedings of the 28th ACM International Conference on Supercomputing

View full text Add to dashboard Cite

show abstract

“…Nessie [LOKR11, LOK12], short for NEtwork Scalable Service InterfacE, is a framework for developing parallel, application‐specific data Services for HPC systems. These Services include checkpointing, interactive visualization, network traffic analysis, and in transit analysis.…”

Section: In Situ History and Surveymentioning

confidence: 99%

In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms

Bauer

Abbasi

Ahrens

et al. 2016

Computer Graphics Forum

144

View full text Add to dashboard Cite

The considerable interest in the high performance computing (HPC) community regarding analyzing and visualization data without first writing to disk, i.e., in situ processing, is due to several factors. First is an I/O cost savings, where data is analyzed/visualized while being generated, without first storing to a filesystem. Second is the potential for increased accuracy, where fine temporal sampling of transient analysis might expose some complex behavior missed in coarse temporal sampling. Third is the ability to use all available resources, CPU's and accelerators, in the computation of analysis products. This STAR paper brings together researchers, developers and practitioners using in situ methods in extreme-scale HPC with the goal to present existing methods, infrastructures, and a range of computational science and engineering applications using in situ analysis and visualization.

show abstract

“…It does not require the full features of traditional HPC runtimes, but does require mechanisms for event routing and transfer, proper placement in a network topology, and signals for event activation. Examples include topological analysis for the S3D combustion simulation code [6], and streaming network analysis [20].…”

Section: Motivating Use Casesmentioning

confidence: 99%

Hobbes

Brightwell

Oldfield

Maccabe

et al. 2013

Proceedings of the 3rd International Workshop on Runtime and Operating Systems for Supercomputers

View full text Add to dashboard Cite

This paper describes our vision for Hobbes, an operating system and runtime (OS/R) framework for extreme-scale systems. The Hobbes design explicitly supports application composition, which is emerging as a key approach for applications to address scalability and power concerns anticipated with coming extreme-scale architectures. We make use of virtualization technologies to provide the flexibility to support requirements of application components for different node-level operating systems and runtimes, as well as different mappings of the components onto the hardware. We describe the architecture of the Hobbes OS/R, how we will address the cross-cutting concerns of power/energy, scheduling of massive levels of parallelism, and resilience. We also outline how the "users" of the OS/R (programming models, applications, and tools) influence the design.

show abstract

Experiences Applying Data Staging Technology in Unconventional Ways

Cited by 5 publications

References 18 publications

Evaluation of methods to integrate analysis into a large-scale shock shock physics code

Evaluation of methods to integrate analysis into a large-scale shock shock physics code

In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms

Hobbes

Contact Info

Product

Resources

About