Improving Generic Non-contiguous File Access for MPI-IO

Worringen, Joachim; Träff, Jesper Larsson; Ritzdorf, Hubert

doi:10.1007/978-3-540-39924-7_44

Cited by 7 publications

(7 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It could be shown that listless I/O is prepared to perform well for all cases of non-contiguous I/O. In a companion paper [17], we have shown that listless I/O is also effective on scalar cluster architectures, increasing the bandwidth about one order of magnitude.…”

Section: Discussionmentioning

confidence: 88%

“…Although originally designed to take advantage of the hardware support for gather-scatter operations, flattening-on-the-fly has been extended to be efficient on scalar architectures, too, by better exploiting data locality. This is borne out by the implementation of listless-io for PC architectures described in [17].…”

Section: Flattening On the Flymentioning

confidence: 99%

“…[1,2,7,9,11,12,13,17]. Most work is about the implementation of the well-known, portable open-source ROMIO implementation of MPI-IO [12].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Fast Parallel Non-Contiguous File Access

Worringen

Träff

Ritzdorf

2003

Proceedings of the 2003 ACM/IEEE Conference on Supercomputing

Self Cite

View full text Add to dashboard Cite

Many applications of parallel I/O perform non-contiguous file accesses: instead of accessing a single (large) block of data in a file, a number of (smaller) blocks of data scattered throughout the file needs to be accessed in each logical I/O operation. However, only few file system interfaces directly support this kind of non-contiguous file access. In contrast, the most commonly used parallel programming interface, MPI, incorporates a flexible model of parallel I/O through its MPI-IO interface. With MPI-IO, arbitrary non-contiguous file accesses are supported in a uniform fashion by the use of derived MPI datatypes set up by the user to reflect the desired I/O pattern.Despite a considerable amount of recent work in this area, current MPI-IO implementations suffer from low performance of such non-contiguous accesses when compared to the performance of the storage system for contiguous accesses. In this paper we analyze an important bottleneck in the efficient handling of non-contiguous access patterns in current implementations of MPI-IO. We present a new technique, termed listless I/O, that can be incorporated into MPI-IO implementations like the well-known ROMIO implementation, and completely eliminates this bottleneck. We have implemented the technique in MPI/SX, the MPI implementation for the NEC SX-series of parallel vector computers. Results with a synthetic benchmark and an application kernel show that listless I/O is able to increase the bandwidth for non-contiguous file access by sometimes more than a factor of 500 when compared to the traditional approach.

show abstract

Section: Discussionmentioning

confidence: 88%

Section: Flattening On the Flymentioning

confidence: 99%

See 1 more Smart Citation

Fast Parallel Non-Contiguous File Access

Worringen

Träff

Ritzdorf

2003

Proceedings of the 2003 ACM/IEEE Conference on Supercomputing

Self Cite

View full text Add to dashboard Cite

show abstract

“…After declaring a view on a file, a process may see and access non-contiguous regions of the file in a contiguous manner. In [21], a new datatype handling functionality is introduced. It improves noncontiguous data access management, with the result of increasing the data throughput.…”

Section: Related Workmentioning

confidence: 99%

A collective I/O implementation based on inspector–executor paradigm

et al. 2008

View full text Add to dashboard Cite

In this paper, we present a novel multiple phase I/O collective technique for generic block-cyclic distributions. The I/O technique is divided into two stages: inspector and executor. During the inspector stage, the communication pattern is computed and the required datatypes are automatically generated. This information is used during the executor stage in performing the communication and file accesses. The two stages are decoupled, so that for repetitive file access patterns, the computations from the inspector stage can be performed once and reused several times by the executor. This strategy allows to amortize the inspector cost over several I/O operations. In this paper, we evaluate the performance of multiple phase I/O collective technique and we compare it with other state of the art approaches. Experimental results show that for small access granularities, our method outperforms in the large majority of cases other parallel I/O optimizations techniques.

show abstract

“…Operating on the datatypes directly instead of "flattening" the datatypes or entire accesses into offset/length pairs can also ease memory requirements of representing and storing data layout descriptions [16]. There is a threshold where storing datatypes does require more memory than storing the flat offset/length pairs representing the datatypes.…”

Section: Related Workmentioning

confidence: 99%