On implementing MPI-IO portably and with high performance

Abstract

We discuss the issues involved in implementing MPI-IO portably on multiple machines and le systems and also achieving high performance. One way t o i m p l e m e n t MPI-IO portably is to implement it on top of the basic Unix I/O functions (open, lseek, read, write, a n d close), which a r e themselves portable. We argue that this approach has limitations in both functionality and performance. We instead advocate an implementation approach t h a t c o m bines a large portion of portable code and a small portion of code that is optimized separately for di erent m a c hines and le systems. We h a ve used such an approach to develop a high-performance, portable MPI-IO implementation, called ROMIO.In addition to basic I/O functionality, w e consider the issues of supporting other MPI-IO features, such as 64-bit le sizes, noncontiguous accesses, collective I/O, asynchronous I/O, consistency and atomicity s e m a n tics, user-supplied hints, shared le pointers, portable data representation, le preallocation, and some miscellaneous features. We describe how w e implemented each of these features on various machines and le systems. The machines we consider are the HP Exemplar, IBM SP, Intel Paragon, NEC SX-4, SGI Origin2000, and networks of workstations and the le systems we consider are HP HFS, IBM PIOFS, Intel PFS, NEC SFS, SGI XFS, NFS, and any general Unix le system (UFS).We also present our thoughts on how a le system can be designed to better support MPI-IO. We provide a list of features desired from a le system that would help in implementing MPI-IO correctly and with high performance.

Towards a High Performance Implementation of MPI-IO on the Lustre File System

“…In [19], an objectbased caching system is shown to improve the performance of ROMIO on the FLASH I/O benchmark [5]. There has also been significant research related to improving the performance of non-contiguous I/O requests in ROMIO [24], independent writes [17], and workarounds to the POSIX API (e.g., data sieving [22], two-phase I/O [23]). Our research is similarly focused on improving the performance of ROMIO in a particular file system environment.…”

“…First, Lustre exports only the POSIX file system API, which was not designed for a parallel I/O environment and provides little support for parallel I/O optimizations. This has led to the development of approaches (or "workarounds") that can circumvent (most of) the performance problems inherent in POSIX-based file systems, such that the performance of MPI-IO in such environments can be significantly improved (e.g., two-phase I/O [22,23], data-sieving [25], DataType I/O [11]). The second problem is that the assumptions upon which most of these optimizations are based do not hold in a Lustre environment.…”

“…ADIO implements the file system dependent features, and is thus implemented separately for each file system (see Figure 1). ROMIO implements the collective I/O operations using a technique termed two-phase I/O [23,25]. Consider a collective write operation.…”

Exploiting Shared Memory to Improve Parallel I/O Performance

“…Via this facility, an application using MPI-IO may issue I/O operations that are non-contiguous in memory and/or in a file [1]. Previous work has explored optimizing these operations via data sieving [2], the two-phase collective optimization [2], list I/O [3], and datatype I/O [4], primarily in the context of commodity clusters. The first two optimizations are broadly available through the open source ROMIO implementation distributed as part of MPICH2 [5].…”

Open Issues in MPI Implementation

“…The most commonly used implementation of MPI-IO is ROMIO [20,24]. To our knowledge, almost all MPI implementations, except IBM's MPI for the SP, use ROMIO as the basis for MPI-IO.…”