Parallel I/O prefetching using MPI file caching and I/O signatures

Abstract

-Parallel I/O prefetching is considered to be effective in improving I/O performance. However, the effectiveness depends on determining patterns among future I/O accesses swiftly and fetching data in time, which is difficult to achieve in general. In this study, we propose an I/O signature-based prefetching strategy. The idea is to use a predetermined I/O signature of an application to guide prefetching. To put this idea to work, we first derived a classification of patterns and introduced a simple and effective signature notation to represent patterns. We then developed a toolkit to trace and generate I/O signatures automatically. Finally, we designed and implemented a thread-based client-side collective prefetching cache layer for MPI-IO library to support prefetching. A prefetching thread reads I/O signatures of an application and adjusts them by observing I/O accesses at runtime. Experimental results show that the proposed prefetching method improves I/O performance significantly for applications with complex patterns.

Boosting Application-Specific Parallel I/O Optimization Using IOSIG

“…The second is to perform offline analysis on the trace files generated in the previous step, to produce data access pattern information. The trace analyzer produces I/O characteristics in the form represented in I/O signatures defined in our previous work [14]. The produced I/O signatures are easy to be read by humans or automatic optimization tools for capturing I/O behavior and for improving data access performance.…”

“…The local patterns explain how each process accesses data and the global patterns provide an overview of the I/O characteristics of an application. We have classified local file access patterns in [14]. In this section, we introduce local and global access patterns and I/O signatures and describe the implementation of the trace analyzer.…”

Omnisc'IO: A Grammar-Based Approach to Spatial and Temporal I/O Patterns Prediction

“…These approaches often focus exclusively on either spatial or temporal I/O behaviors. Furthermore, they require a large number of observations to accomplish good prediction; hence, they either need long execution time (several runs in some cases) [9] or are doomed to offline trace-based training [10] in order to converge. Our work addresses the limitations of current prediction systems and takes a step toward intelligent I/O management of HPC applications in next-generation post-petascale supercomputers [11] that is capable of run-time analysis and adaptation to the I/O behavior of applications.…”

“…Others have already demonstrated the benefits of applying I/O predications to enhance the performance in each of these techniques [8], [6], [9], [10]. We focus our study on the prediction capabilities of Omnisc'IO.…”

Orthrus: A Framework for Implementing Efficient Collective I/O in Multi-core Clusters

“…Two collective I/O calls are deemed to have the same pattern and comparable performance results only when their signatures are the same. I/O signature-based pattern identification has been effectively used in previous works, for example in data prefetching [4].…”