I/O acceleration with pattern detection

He, Jie; Bent, John; Torres, Aaron; Grider, Gary; Gibson, Garth A.; Maltzahn, Carlos; Sun, Xian-He

doi:10.1145/2493123.2462909

Cited by 48 publications

(14 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Closer to our approach is the work by He et al [7], who propose an approach to spatial I/O pattern detection to improve metadata indexing in PLFS. Their approach considers a sequence of (offset,size) access parameters and tries to find repetitive patterns in the differences (delta) between consecutive accesses, using a method inspired by the LZ77 sliding window algorithm.…”

Section: ) Spatial Predictionsmentioning

confidence: 99%

“…The effectiveness of such techniques strongly depends on a certain level of knowledge of the I/O access patterns: prefetching and caching indeed require the location of future accesses (i.e., spatial behavior), while I/O scheduling leverages estimations of I/O requests interarrival time (i.e., temporal behavior). The key challenges inherent in these techniques include the proper comprehension and exploitation of the application's I/O behavior within the I/O stack itself [7], [5]. Hence, modeling and predicting the applications I/O behavior are of utmost importance.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Omnisc'IO: A Grammar-Based Approach to Spatial and Temporal I/O Patterns Prediction

Dorier

Ibrahim²,

Antoniu³

et al. 2014

SC14: International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

Abstract-The increasing gap between the computation performance of post-petascale machines and the performance of their I/O subsystem has motivated many I/O optimizations including prefetching, caching, and scheduling techniques. In order to further improve these techniques, modeling and predicting spatial and temporal I/O patterns of HPC applications as they run has became crucial.In this paper we present Omnisc'IO, an approach that builds a grammar-based model of the I/O behavior of HPC applications and uses it to predict when future I/O operations will occur, and where and how much data will be accessed. Omnisc'IO is transparently integrated into the POSIX and MPI I/O stacks and does not require any modification in applications or higherlevel I/O libraries. It works without any prior knowledge of the application and converges to accurate predictions within a couple of iterations only. Its implementation is efficient in both computation time and memory footprint.

show abstract

Section: ) Spatial Predictionsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Omnisc'IO: A Grammar-Based Approach to Spatial and Temporal I/O Patterns Prediction

Dorier

Ibrahim²,

Antoniu³

et al. 2014

SC14: International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

show abstract

“…He et al [33] propose an approach to improve read operations with the PLFS library. Information from trace files -generated while writing data -is used to reconstruct the high-level data structures used by the application (through MPI-IO or HDF5).…”

Section: Hybrid Runtime + Postmortem Approachesmentioning

confidence: 99%

A Checkpoint of Research on Parallel I/O for High-Performance Computing

et al. 2018

View full text Add to dashboard Cite

We present a comprehensive survey on parallel I/O in the high performance computing (HPC) context. This is an important field for HPC because of the historic gap between processing power and storage latencies, which causes applications performance to be impaired when accessing or generating large amounts of data. As the available processing power and amount of data increase, I/O remains a central issue for the scientific community. In this survey, we focus on a traditional I/O stack, with a POSIX parallel file system. We present background concepts everyone could benefit from. Moreover, through the comprehensive study of publications from the most important conferences and journals in a five-year time window, we discuss the state of the art of I/O optimization approaches, access pattern extraction techniques, and performance modeling, in addition to general aspects of parallel I/O research. Through this approach, we aim at identifying the general characteristics of the field and the main current and future research topics.

show abstract

“…Another recent study introduced methods to automatically identify I/O patterns at the block level and utilized them to manage the metadata indexing and support data prefetching in physics and geoscience applications [38]. Statistical analysis of NetCDF scientific datasets was successfully explored over the Hadoop platform developed for big data applications [10].…”

Section: Big Data + Hpc Integrated Sw Stackmentioning

confidence: 99%

On Processing Extreme Data

Petcu¹,

Iuhasz²,

Pop³

et al. 2016

SCPE

View full text Add to dashboard Cite

Abstract. Extreme Data is an incarnation of Big Data concept distinguished by the massive amounts of data that must be queried, communicated and analyzed in near real-time by using a very large number of memory or storage elements and exascale computing systems. Immediate examples are the scientific data produced at a rate of hundreds of gigabits-per-second that must be stored, filtered and analyzed, the millions of images per day that must be analyzed in parallel, the one billion of social data posts queried in real-time on an in-memory components database. Traditional disks or commercial storage nowadays cannot handle the extreme scale of such application data.Following the need of improvement of current concepts and technologies, we focus in this paper on the needs of data intensive applications running on systems composed of up to millions of computing elements (exascale systems). We propose in this paper a methodology to advance the state-of-the-art. The starting point is the definition of new programming paradigms, APIs, runtime tools and methodologies for expressing data-intensive tasks on exascale systems. This will pave the way for the exploitation of massive parallelism over a simplified model of the system architecture, thus promoting high performance and efficiency, offering powerful operations and mechanisms for processing extreme data sources at high speed and/or real time.

show abstract

I/O acceleration with pattern detection

Cited by 48 publications

References 31 publications

Omnisc'IO: A Grammar-Based Approach to Spatial and Temporal I/O Patterns Prediction

Omnisc'IO: A Grammar-Based Approach to Spatial and Temporal I/O Patterns Prediction

A Checkpoint of Research on Parallel I/O for High-Performance Computing

On Processing Extreme Data

Contact Info

Product

Resources

About