Managing Variability in the IO Performance of Petascale Storage Systems

Lofstead, Jay; Zheng, Fang; Liu, Qing; Klasky, Scott; Oldfield, Ron A.; Kordenbrock, Todd; Schwan, Karsten; Wolf, Matthew

doi:10.1109/sc.2010.32

Cited by 130 publications

(88 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, it can alleviate lock contention issues that arise in parallel file system writes [11]. Second, in a similar method to [18], we found that pinning files to storage targets in a round-robin manner provided an efficient method for managing variability in writes. Since users define their output in Protocol Buffer format, the data can easily be parsed afterwards serially or in parallel.…”

mentioning

confidence: 81%

Simplified parallel domain traversal

Kendall

Wang

Allen

et al. 2011

Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

Many data-intensive scientific analysis techniques require global domain traversal, which over the years has been a bottleneck for efficient parallelization across distributedmemory architectures. Inspired by MapReduce and other simplified parallel programming approaches, we have designed DStep, a flexible system that greatly simplifies efficient parallelization of domain traversal techniques at scale. In order to deliver both simplicity to users as well as scalability on HPC platforms, we introduce a novel two-tiered communication architecture for managing and exploiting asynchronous communication loads. We also integrate our design with advanced parallel I/O techniques that operate directly on native simulation output. We demonstrate DStep by performing teleconnection analysis across ensemble runs of terascale atmospheric CO2 and climate data, and we show scalability results on up to 65,536 IBM BlueGene/P cores.

show abstract

mentioning

confidence: 81%

Simplified parallel domain traversal

Kendall

Wang

Allen

et al. 2011

Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

show abstract

“…They use information about applications access patterns to dynamically decide between three scheduling strategies, seeking to optimize a given metric. Lofstead et al [62] propose an adaptive I/O method, implemented in their ADIOS middleware, which monitors the file system performance and balances the workload accordingly.…”

Section: I/o Schedulingmentioning

confidence: 99%

A Checkpoint of Research on Parallel I/O for High-Performance Computing

et al. 2018

View full text Add to dashboard Cite

We present a comprehensive survey on parallel I/O in the high performance computing (HPC) context. This is an important field for HPC because of the historic gap between processing power and storage latencies, which causes applications performance to be impaired when accessing or generating large amounts of data. As the available processing power and amount of data increase, I/O remains a central issue for the scientific community. In this survey, we focus on a traditional I/O stack, with a POSIX parallel file system. We present background concepts everyone could benefit from. Moreover, through the comprehensive study of publications from the most important conferences and journals in a five-year time window, we discuss the state of the art of I/O optimization approaches, access pattern extraction techniques, and performance modeling, in addition to general aspects of parallel I/O research. Through this approach, we aim at identifying the general characteristics of the field and the main current and future research topics.

show abstract

“…Based on the observation that within a single application, processes already interfere with each other, Lofstead et al [25] propose an adaptive I/O approach in which the processes of an application are gathered in groups. Each group writes in a particular storage target, and one process in each group is chosen to coordinate the accesses issued by all the other in the group.…”

Section: B Application-side I/o Schedulingmentioning

confidence: 99%

CALCioM: Mitigating I/O Interference in HPC Systems through Cross-Application Coordination

Dorier

Antoniu

Ross

et al. 2014

2014 IEEE 28th International Parallel and Distributed Processing Symposium

100

View full text Add to dashboard Cite

Abstract-Unmatched computation and storage performance in new HPC systems have led to a plethora of I/O optimizations ranging from application-side collective I/O to network and disk-level request scheduling on the file system side. As we deal with ever larger machines, the interference produced by multiple applications accessing a shared parallel file system in a concurrent manner becomes a major problem. Interference often breaks single-application I/O optimizations, dramatically degrading application I/O performance and, as a result, lowering machine wide efficiency.This paper focuses on CALCioM, a framework that aims to mitigate I/O interference through the dynamic selection of appropriate scheduling policies. CALCioM allows several applications running on a supercomputer to communicate and coordinate their I/O strategy in order to avoid interfering with one another. In this work, we examine four I/O strategies that can be accommodated in this framework: serializing, interrupting, interfering and coordinating. Experiments on Argonne's BG/P Surveyor machine and on several clusters of the French Grid'5000 show how CALCioM can be used to efficiently and transparently improve the scheduling strategy between two otherwise interfering applications, given specified metrics of machine wide efficiency.

show abstract

Managing Variability in the IO Performance of Petascale Storage Systems

Cited by 130 publications

References 31 publications

Simplified parallel domain traversal

Simplified parallel domain traversal

A Checkpoint of Research on Parallel I/O for High-Performance Computing

CALCioM: Mitigating I/O Interference in HPC Systems through Cross-Application Coordination

Contact Info

Product

Resources

About