Access Patterns and Performance Behaviors of Multi-layer Supercomputer I/O Subsystems under Production Load

Abstract

Scientific computing workloads at HPC facilities have been shifting from traditional numerical simulations to AI/ML applications for training and inference while processing and producing everincreasing amounts of scientific data. To address the growing need for increased storage capacity, lower access latency, and higher bandwidth, emerging technologies such as non-volatile memory are integrated into supercomputer I/O subsystems. With these emerging trends, we need a better understanding of the multilayer supercomputer I/O systems and ways to use these subsystems efficiently. In this work, we study the I/O access patterns and performance characteristics of two representative supercomputer I/O subsystems. Through an extensive analysis of year-long I/O logs on each system, we report new observations in I/O reads and writes, unbalanced use of storage system layers, and new trends in user behaviors at the HPC I/O middleware stack.

Drishti: Guiding End-Users in the I/O Optimization Journey

“…10) High STDIO Usage: Excessive use of STDIO for HPC applications might harm I/O performance [39], [40]. Highlevel interfaces such as MPI-IO are a better alternative.…”