Efficient Algorithms for Block-Cyclic Redistribution of Arrays

Abstract

The block-cyclic data distribution is commonly used to organize array elements over the processors of a coarse-grained distributed memory parallel computer. In many scientific applications, the data layout must be reorganized at run-time in order to enhance locality and reduce remote memory access overheads. In this paper we present a general framework for developing array redistribution algorithms. Using this framework, we have developed efficient algorithms that redistribute an array from one block-cyclic layout to another.Block-cyclic redistribution consists of index set computation, wherein the destination locations for individual data blocks are calculated, and data communication, wherein these blocks are exchanged between processors. The framework treats both these operations in a uniform and integrated way. We have developed efficient and distributed algorithms for index set computation that do not require any interprocessor communication. To perform data communication in a conflict-free manner, we have developed direct, indirect, and hybrid algorithms. In the direct algorithm, a data block is transferred directly to its destination processor. In an indirect algorithm, data blocks are moved from source to destination processors through intermediate relay processors. The hybrid algorithm is a combination of the direct and indirect algorithms.Our framework is based on a generalized circulant matrix formalism of the redistribution problem and a general purpose distributed memory model of the parallel machine. Our algorithms sustain excellent performance over a wide range of problem and machine parameters. We have implemented our algorithms using MPI, to allow for easy portability across different HPC platforms. Experimental results on the IBM SP-2 and the Cray T3D show superior performance over previous approaches. When the block size of the cyclic data layout changes by a factor of K , the redistribution can be performed in O(log K ) communication steps. This is true even when K is a prime number. In contrast, previous approaches take O(K ) communication steps for redistribution.Our framework can be used for developing scalable redistribution libraries, for efficiently implementing parallelizing compiler directives, and for developing parallel algorithms for various applications. Redistribution algorithms are especially useful in signal processing applications, where the data access patterns change significantly between computational phases. They are also necessary in linear algebra programs, to perform matrix transpose operations.

Scheduling contention-free irregular redistributions in parallelizing compilers

“…Examples are the processor mapping techniques [4,10,12] for minimizing data transmission overheads, the multiphase redistribution strategy [11] for reducing message startup cost, the communication scheduling approaches [2,7,13,20] for avoiding node contention and the strip mining approach [18] for overlapping communication and computational overheads.…”

Efficient Multidimensional Data Redistribution for Resizable Parallel Computations

“…The generalized BCC uses uses bipartite matching approach for data redistribution. Lim et al [8] developed a redistribution framework that could redistribute one-dimensional array from one block-cyclic scheme to another on the same processor set using a generalized circulant matrix formalism. Their algorithm applies row and column transformations on the communication schedule matrix to generate a conflict-free schedule.…”

Parallel Implementation of a Class of Adaptive Signal Processing Applications

“…This approach incurs minimum data transmission cost but communication start-up cost can be high. To minimize the start-up cost, indirect schedules [16]- [18] can be used. In this approach, data blocks are sent to their destination through intermediate "relay" nodes.…”