CALCioM: Mitigating I/O Interference in HPC Systems through Cross-Application Coordination

Dorier, Matthieu; Antoniu, Gabriel; Ross, Robert; Kimpe, Dries; Ibrahim, Shadi

doi:10.1109/ipdps.2014.27

Cited by 95 publications

(87 citation statements)

References 32 publications

(28 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our previous work [4] we advocated for cross-application coordination to mitigate I/O interference. While the application user was required to explicitly instrument an application to expose its I/O patterns to other applications, the spatial and temporal I/O predictions presented in the present work can be leveraged to remove the need for this instrumentation and thus offer transparent cross-application I/O scheduling.…”

Section: ) Spatial Predictionsmentioning

confidence: 99%

“…To alleviate the performance penalty of the I/O bottleneck in petascale systems, researchers have explored techniques such as prefetching, caching, and scheduling [4], [5], [6] to improve I/O performance. The effectiveness of such techniques strongly depends on a certain level of knowledge of the I/O access patterns: prefetching and caching indeed require the location of future accesses (i.e., spatial behavior), while I/O scheduling leverages estimations of I/O requests interarrival time (i.e., temporal behavior).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Omnisc'IO: A Grammar-Based Approach to Spatial and Temporal I/O Patterns Prediction

Dorier

Ibrahim²,

Antoniu³

et al. 2014

SC14: International Conference for High Performance Computing, Networking, Storage and Analysis

Self Cite

View full text Add to dashboard Cite

Abstract-The increasing gap between the computation performance of post-petascale machines and the performance of their I/O subsystem has motivated many I/O optimizations including prefetching, caching, and scheduling techniques. In order to further improve these techniques, modeling and predicting spatial and temporal I/O patterns of HPC applications as they run has became crucial.In this paper we present Omnisc'IO, an approach that builds a grammar-based model of the I/O behavior of HPC applications and uses it to predict when future I/O operations will occur, and where and how much data will be accessed. Omnisc'IO is transparently integrated into the POSIX and MPI I/O stacks and does not require any modification in applications or higherlevel I/O libraries. It works without any prior knowledge of the application and converges to accurate predictions within a couple of iterations only. Its implementation is efficient in both computation time and memory footprint.

show abstract

Section: ) Spatial Predictionsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Omnisc'IO: A Grammar-Based Approach to Spatial and Temporal I/O Patterns Prediction

Dorier

Ibrahim²,

Antoniu³

et al. 2014

SC14: International Conference for High Performance Computing, Networking, Storage and Analysis

Self Cite

View full text Add to dashboard Cite

show abstract

“…These portions are then replicated to other servers to decrease the concurrency. Dorier, Antoniu, and Ross [23] propose a client-side cross-application coordination strategy. They use information about applications access patterns to dynamically decide between three scheduling strategies, seeking to optimize a given metric.…”

Section: I/o Schedulingmentioning

confidence: 99%

A Checkpoint of Research on Parallel I/O for High-Performance Computing

et al. 2018

View full text Add to dashboard Cite

We present a comprehensive survey on parallel I/O in the high performance computing (HPC) context. This is an important field for HPC because of the historic gap between processing power and storage latencies, which causes applications performance to be impaired when accessing or generating large amounts of data. As the available processing power and amount of data increase, I/O remains a central issue for the scientific community. In this survey, we focus on a traditional I/O stack, with a POSIX parallel file system. We present background concepts everyone could benefit from. Moreover, through the comprehensive study of publications from the most important conferences and journals in a five-year time window, we discuss the state of the art of I/O optimization approaches, access pattern extraction techniques, and performance modeling, in addition to general aspects of parallel I/O research. Through this approach, we aim at identifying the general characteristics of the field and the main current and future research topics.

show abstract

“…Additionally, research into the layer above the parallel file system is beginning to uncover methods of orchestrating I/O between applications [16]. This type of high-level coordination can assist in managing shared resources such as network links and I/O gateways and is complementary to an understanding of the storage data layout itself.…”

Section: Trends and Requirementsmentioning

confidence: 99%

Trends in Data Locality Abstractions for HPC Systems

Unat

Dubey

Hoefler

et al. 2017

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

Abstract-The cost of data movement has always been an important concern in high performance computing (HPC) systems. It has now become the dominant factor in terms of both energy consumption and performance. Support for expression of data locality has been explored in the past, but those efforts have had only modest success in being adopted in HPC applications for various reasons. However, with the increasing complexity of the memory hierarchy and higher parallelism in emerging HPC systems, locality management has acquired a new urgency. Developers can no longer limit themselves to low-level solutions and ignore the potential for productivity and performance portability obtained by using locality abstractions. Fortunately, the trend emerging in recent literature on the topic alleviates many of the concerns that got in the way of their adoption by application developers. Data locality abstractions are available in the forms of libraries, data structures, languages and runtime systems; a common theme is increasing productivity without sacrificing performance. This paper examines these trends and identifies commonalities that can combine various locality concepts to develop a comprehensive approach to expressing and managing data locality on future large-scale high-performance computing systems.

show abstract

CALCioM: Mitigating I/O Interference in HPC Systems through Cross-Application Coordination

Cited by 95 publications

References 32 publications

Omnisc'IO: A Grammar-Based Approach to Spatial and Temporal I/O Patterns Prediction

Omnisc'IO: A Grammar-Based Approach to Spatial and Temporal I/O Patterns Prediction

A Checkpoint of Research on Parallel I/O for High-Performance Computing

Trends in Data Locality Abstractions for HPC Systems

Contact Info

Product

Resources

About