Hierarchical I/O Scheduling for Collective I/O

Abstract

The non-contiguous access pattern of many scientific applications results in a large number of I/O requests, which can seriously limit the data-access performance. Collective I/O has been widely used to address this issue. However, the performance of collective I/O could be dramatically degraded in today's high-performance computing system due to the increasing shuffle cost caused by highly concurrent data accesses. This situation tends to be even worse as many applications become more and more data intensive. Previous research has primarily focused on optimizing I/O access cost in collective I/O but largely ignored the shuffle cost involved. In this study, we propose a new hierarchical I/O scheduling (HIO) algorithm to address the increasing shuffle cost in collective I/O. The fundamental idea is to schedule applications' I/O requests based on a shuffle cost analysis to achieve the optimal overall performance, instead of achieving optimal I/O accesses only. The algorithm is currently evaluated with the MPICH2 and PVFS2. Both theoretical analysis and experimental tests show that the proposed hierarchical I/O scheduling has a potential in addressing the degraded performance issue of collective I/O with highly concurrent accesses.

Capturing Periodic I/O Using Frequency Techniques

“…More general characterization efforts usually focus on aspects such as spatiality and request size [49], [50], using information from MPI-IO [51]- [53], ML-based methods [43], etc. In contrast, FTIO focuses on the temporal behavior (specifically on the periodicity), and hence is also complementary to those.…”

Supporting dynamic allocation of heterogeneous storage resources on HPC systems

“…That is, they offer algorithms to orchestrate individual I/Os or I/O phases of concurrently executing applications. They can be specifically directed at the application level, [15][16][17][18] the I/O forwarding level, 19,20 or can be integrated into middlewares in charge of offloading such I/O requests. 21 Another common approach consists of considering the availability of I/O resources as a constraint when placing the scheduled tasks onto the compute nodes, without subsequently interfering with the way the applications do I/Os.…”