On the Root Causes of Cross-Application I/O Interference in HPC Storage Systems

Abstract

As we move toward the exascale era, performance variability in HPC systems remains a challenge. I/O interference, a major cause of this variability, is becoming more important every day with the growing number of concurrent applications that share larger machines. Earlier research efforts on mitigating I/O interference focus on a single potential cause of interference (e.g., the network). Yet the root causes of I/O interference can be diverse. In this work, we conduct an extensive experimental campaign to explore the various root causes of I/O interference in HPC storage systems. We use microbenchmarks on the Grid'5000 testbed to evaluate how the applications' access pattern, the network components, the file system's configuration, and the backend storage devices influence I/O interference. Our studies reveal that in many situations interference is a result of bad flow control in the I/O path, rather than being caused by some single bottleneck in one of its components. We further show that interferencefree behavior is not necessarily a sign of optimal performance. To the best of our knowledge, our work provides the first deep insight into the role of each of the potential root causes of interference and their interplay. Our findings can help developers and platform owners improve I/O performance and motivate further research addressing the problem across all components of the I/O stack.

ONFS: a hierarchical hybrid file system based on memory, SSD, and HDD for high performance computers

“…The capacity of a single HDD-based storage system has reached dozens of PBs (NERSC, 2017b). Although redundant array of independent disks (RAID) has been leveraged to improve the performance of storage systems, HDD's costly overhead of disk-head movements prevents it from meeting the extreme amount of input/output (I/O) and low I/O latency requirements of future exascale systems (Yildiz et al, 2016).…”

How File-access Patterns Influence the Degree of I/O Interference between Cluster Applications

“…Yildiz et al studied the root cause of inter-application I/O interference in HPC storage systems by comparing the impact of different factors [19]. They found that bad flow of control in the I/O path caused interference in most cases.…”

Cross‐layer coordination in the I/O software stack of extreme‐scale systems

“…HDD is known for its significant performance degradation under concurrent workloads. The interference stems from the additional disk‐head movements produced by interleaved requests to distinct file regions . The total bandwidth a storage node can provide is greatly cut down when there is competition.…”

“…Many works in this area() have been dedicated to mitigate the increasingly serious problem of cross‐application I/O interference . An I/O orchestration mechanism named TRIO is proposed to coordinate bursty writes of checkpoint data on I/O nodes for better sequential write traffic to storage nodes.…”

“…Many works in this area 32,33 have been dedicated to mitigate the increasingly serious problem of cross-application I/O interference. 34 Delegation-based I/O 36 is an MPI based mechanism that uses delegate processes instead of I/O nodes to forward application requests. Different delegate processes communicate with each other before writing, and shuffle their data to merge small noncontiguous data requests into large sequential ones.…”