A collective I/O implementation based on inspector–executor paradigm

Abstract

In this paper, we present a novel multiple phase I/O collective technique for generic block-cyclic distributions. The I/O technique is divided into two stages: inspector and executor. During the inspector stage, the communication pattern is computed and the required datatypes are automatically generated. This information is used during the executor stage in performing the communication and file accesses. The two stages are decoupled, so that for repetitive file access patterns, the computations from the inspector stage can be performed once and reused several times by the executor. This strategy allows to amortize the inspector cost over several I/O operations. In this paper, we evaluate the performance of multiple phase I/O collective technique and we compare it with other state of the art approaches. Experimental results show that for small access granularities, our method outperforms in the large majority of cases other parallel I/O optimizations techniques.

An implementation of parallel file distribution in an agent hierarchy

“…This work is quite close to ours in terms of the implementation concept, however AgentTeamwork can extend its agent hierarchy to multiple clusters and private network domains over gateways [8]. rearranging these pairs in a logarithmic order [24]. Although AgentTeamwork performs its file distribution in the same order, it differs in using a log x N -based agent tree where x is the maximum number of children each agent can spawn and N is the total number of sentinel agents, (i.e., the number of user processes).…”

Design and Implementation of Automatic Astronomical Spectra Data Processing System

“…In the system, we use "ExecutorService [9]" and "FutureTask" to control the request time. In the first request, 5 seconds can be spend to execute at most.…”

Using Simulation to Validate Performance of MPI(-IO) Implementations

“…These levels are characterized by two orthogonal aspects: contiguous vs. non-contiguous data access, and independent vs. collective calls. Depending on the level, a different set of optimizations can be thought of, for example, two-phase I/O and multiphase-collective I/O [7] aim to improve collective noncontiguous access. An adaptive approach is introduced in [8], which automatically sets hints for collective I/O based on the access pattern, topology and the characteristics of the underlying file system.…”