Automatic parallelization via matrix multiplication

Abstract

Existing work that deals with parallelization of complicated reductions and scans focuses only on formalism and hardly dealt with implementation. To bridge the gap between formalism and implementation, we have integrated parallelization via matrix multiplication into compiler construction. Our framework can deal with complicated loops that existing techniques in compilers cannot parallelize. Moreover, we have sophisticated our framework by developing two sets of techniques. One enhances its capability for parallelization by extracting max-operators automatically, and the other improves the performance of parallelized programs by eliminating redundancy. We have also implemented our framework and techniques as a parallelizer in a compiler. Experiments on examples that existing compilers cannot parallelize have demonstrated the scalability of programs parallelized by our implementation.

On Program Equivalence with Reductions

“…It is possible to translate such program by extracting the true dependences using techniques such as Array Dataflow Analysis [16]. Moreover, if the reductions are not provided, we need to use reduction detection techniques [17,18,19]. Also, there are probably links between the notions introduced in both algorithms, such as our accept domain and their relation R want corresponding to the ("desired correspondence between the iterations of both computations").…”

Enabling prefix sum parallelism pattern for recurrences with principled function reconstruction

“…Benchmarks: Table 3 shows key features of the algorithms used in our experiments. All of these algorithms are collected from previous research efforts in this area [12,20,21]. For other details of these algorithms, readers are referred to [12].…”

“…Figure 3 shows the relationship between the number of thread blocks and speedups of GPU implementations over sequential programs for a set of representative algorithms. These algorithms are collected from the previous research work [12,20,21]. Some key features of these algorithms are shown in Table 3.…”

“…Based on ideas around parallel prefix sum, many efforts have been made over the last decade to parallelize a class of seemingly sequential loops that involve recurrences [3,[7][8][9]21]. These loops are hard to parallelize because there are strong dependencies across the iterations.…”

Safe Automated Refactoring for Intelligent Parallelization of Java 8 Streams

“…The general problem of full automatic parallelization by compilers is extremely complex and remains a grand challenge [55]. Many attempt to solve it in only certain contexts, e.g., for divide and conquer [56], recursive functions [57], distributed architectures [58], graphics processing [59], matrix manipulation [60], asking the developer for assistance [61], and speculative strategies [62]. Our approach focuses on MapReduce-style code over native data containers in a shared memory space using a mainstream programming languages, which may be more amenable to parallelization due to more explicit data dependencies [16].…”