A unified framework for optimizing communication in data-parallel programs

Abstract

This paper presents a framework, based on global array data-ow analysis, to reduce communication costs in a program being compiled for a distributed memory machine. We introduce available section descriptor, a novel representation of communication involving array sections. This representation allows us to apply techniques for partial redundancy elimination to obtain powerful communication optimizations. With a single framework, we are able to capture optimizations like (i) vectorizing communication, (ii) eliminating communication that is redundant on any control ow path, (iii) reducing the amount of data being communicated, (iv) reducing the number of processors to which data must be communicated, and (v) moving communication earlier to hide latency, and to subsume previous communication. We show that the bidirectional problem of eliminating partial redundancies can be decomposed into simpler unidirectional problems even in the context of an array section representation, which makes the analysis procedure more e cient. We present results from a preliminary implementation of this framework, which are extremely encouraging, and demonstrate the e ectiveness of this analysis in improving the performance of programs.Distributed memory architectures are becoming increasingly popular as a viable and cost-e ective method of building massively parallel computers. However, the absence of global address space, and consequently, the need for explicit message passing among processes makes these machines very di cult to program. This has motivated the design of languages like High Performance Fortran 10], which allow the programmer to write sequential or shared-memory parallel programs that are annotated with directives specifying data decomposition. The compilers for these languages are responsible for partitioning the computation, and generating the communication necessary to fetch values of non-local data referenced by a processor. A number of such prototype compilers have been developed 18,33,23,26,22,25,3,15,28].Since the cost of interprocessor communication is usually orders of magnitude higher than the cost of accessing local data, it is extremely important for the compilers to optimize communication. The most common optimizations include message vectorization 18, 33], using collective communication 14, 23], and overlapping communication with computation 18]. However, most compilers perform little global analysis of the communication requirements across di erent loop nests. This precludes general optimizations, such as redundant communication elimination, or carrying out extra communication inside one loop nest if it subsumes communication required in the next loop nest.This paper presents a framework, based on global array data-ow analysis, to reduce communication in a program. We apply techniques for partial redundancy elimination, discussed in the context of eliminating redundant computation by Morel and Renvoise 24], and later re ned by other researchers 8,20,9]. The conventional approach to data-ow analysis regards...

Integrated range comparison for data-parallel compilation systems

“…It is well-known [9], [10], [11], [12] that the overhead to access nonlocal data from remote processors on distributed memory architectures is commonly orders of magnitude higher than the cost of accessing local data. Communication overhead is, therefore, one of the most important metrics in choosing an appropriate data distribution.…”

Using shared-data localization to reduce the cost of inspector-execution in unified-parallel-C programs

“…When using static analysis for data coalescing in Unified Parallel C [14,8] and High Performance Fortran [13,25], the compiler identifies, through data and control flow analysis, shared accesses to specific threads and creates a single run-time call to access multiple data items from the same thread. However, existing solutions do not completely remove the calls.…”

“…Proposed methods to improve fine-grained communication in PGAS languages include inspector-executor transformation [27,11,5], static coalescing [14,8,25], limited privatization [10,15], and software caching [39]. However, a big hurdle in the code generation of UPC language is that the compiler ends up inserting runtime calls to transform UPC "shared" accesses into requests for data (or actions) to other address partitions.…”

Power aware data and memory management for dynamic applications

“…the SUIF project [40 -42] and the Paradign compiler [43,44]). However, they target an architecture which is very different from ours.…”