Making resonance a common case: A high-performance implementation of collective I/O on parallel file systems

Abstract

Collective I/O is a widely used technique to improve I/O performance in parallel computing. It can be implemented as a client-based or server-based scheme. The client-based implementation is more widely adopted in MPI-IO software such as ROMIO because of its independence from the storage system configuration and its greater portability. However, existing implementations of client-side collective I/O do not take into account the actual pattern of file striping over multiple I/O nodes in the storage system. This can cause a significant number of requests for non-sequential data at I/O nodes, substantially degrading I/O performance.Investigating the surprisingly high I/O throughput achieved when there is an accidental match between a particular request pattern and the data striping pattern on the I/O nodes, we reveal the resonance phenomenon as the cause. Exploiting readily available information on data striping from the metadata server in popular file systems such as PVFS2 and Lustre, we design a new collective I/O implementation technique, resonant I/O, that makes resonance a common case. Resonant I/O rearranges requests from multiple MPI processes according to the presumed data layout on the disks of I/O nodes so that non-sequential access of disk data can be turned into sequential access, significantly improving I/O performance without compromising the independence of a client-based implementation. We have implemented our design in ROMIO. Our experimental results on a small-and medium-scale cluster show that the scheme can increase I/O throughput for some commonly used parallel I/O benchmarks such as mpi-io-test and ior-mpi-io over the existing implementation of ROMIO by up to 157%, with no scenario demonstrating significantly decreased performance.

Orthrus: A Framework for Implementing Efficient Collective I/O in Multi-core Clusters

“…However, if the requests to a disk are issued from different aggregators, there is no way to ensure ascending order unless the involved aggregators synchronize their issuance, which can be excessively expensive, especially in a large-scale system. The ideal scenario in terms of maximizing disk efficiency is to have all requests to a disk (or data node) be issued by one aggregator that can send them in the ascending order according to requested data's offsets in the file [35].…”

“…While maintaining an equal number of aggregators as data nodes allows the data nodes to receive fully sorted sequences of requests, it may limit the I/O bandwidth if the number of data nodes is small. One solution is having multiple aggregators coordinated to access a data node [35].…”

“…Many techniques have been proposed to improve requests' spatial locality by transforming them into large sequential requests, such as data sieving [11] in ROMIO. There are variants of collective I/O recently proposed to take account of on-disk data layout, including Resonant I/O [35] and reducing locking overhead [24] during request transformation. As Orthrus is a framework for dynamically selecting the best performing implementation, these variants are ready to be ported into Orthrus which would then inherit their performance attributes.…”

IOrchestrator: Improving the Performance of Multi-node I/O Systems via Inter-Server Coordination

“…While collective I/O can incur communication overhead because of data exchange among processes, its performance advantage is well recognized, making it one of most popular I/O optimization techniques for MPI programs. Observing that current ROMIO implementations of collective I/O [28] can cause requests to arrive at each data server in an order inconsistent with data placement, resonant I/O was proposed as an enhanced implementation of collective I/O to restore spatial locality [34].…”

Pattern-Direct and Layout-Aware Replication Scheme for Parallel I/O Systems

“…How to optimize I/O performance is elusive, and the optimization is a complex, error-prone, and time-consuming task, especially for applications with complex I/O behaviors. For example, Zhang's work [7] shows that Collective I/O works well in some cases but not in others. Song's work [8] shows that finding the optimal data layout configuration in PVFS2 can be a daunting task.…”