An Optimized Reduction Design to Minimize Atomic Operations in Shared Memory Multiprocessors

Abstract

Reduction operations play a key role in modern massively data parallel computation. However, current implementations in shared memory programming APIs such as OpenMP are often cause of computation bottlenecks due to the high number of atomic operations involved. We propose a reduction design that takes advantage of the coupling with a barrier synchronization to optimize the execution of the reduction. Experimental results show how the number of atomic operations involved is dramatically reduced, which can lead to significant improvement in scaling properties on large numbers of processing elements. We report a speedup of 59.64% on the 312.swim m SPEC OMP2001 benchmark and a speedup of 24.89% on the streamcluster benchmark from the PARSEC suite over the GCC libgomp baseline. IntroductionThe rise of multi-core architectures in recent years has led to the widespread need for parallel software. Given the limited improvements in clock rates, exploiting parallel execution is needed to guarantee performance improvements.Parallelism can be exploited at several levels of granularity, from instruction level parallelism to data parallelism to task parallelism. The OpenMP [1] Application Programming Interface (API) aims at providing an easy-to-use way to program parallel applications at multiple levels of granularity, implemented on top of the C and Fortran languages. Specifically, it targets data and task parallelism by providing directives to identify parallel regions of code and parallel loop constructs.OpenMP also offers a reduction clause to provide some support for recursive array computation, inspired by the reduce or fold constructs of functional languages [6].In functional languages such as Lisp or Haskell, reduce is a higher-order operator that takes as input a binary function f , a list l and an initial value v, and is defined recursively as follows:If the binary function f is associative, it is possible to parallelize the reduce operation, executing it in approximately log2(|l|) steps, where each step i computes a set of intermediate results ti by applying f to pairs of values of ti−1.[Copyright notice will appear here once 'preprint' option is removed.]OpenMP support for reduce-like constructs is limited to associative and commutative binary operators and, in the case of Fortran, intrinsic procedures, which are also associative and commutative functions. Arbitrary functions f are not supported.Reduce-inspired constructs are essential for the expression of data parallelism, as they provide the means to express the extraction of synthetic results from large amounts of data. Recent works in the field of distributed computing [9] show that many data parallel computations can be easily expressed in terms of a reduce-like construct paired with a map-like construct. A map construct essentially allows the execution of a given n-ary function on all the n-uples obtained by taking an element from each of n sequences of equal length.In OpenMP, the parallel loop construct provides the basic data parallelism, replaci...

Combining reduction with synchronization barrier on multi‐core processors

“…In the literature, only Speziale et al 16 propose this method: transporting reduction values on the same cache line as the synchronization flag.…”

“…Figure 7 illustrates this idea. In the literature, only [18] proposes this method: transporting reduction values on the same cache line as the synchronization flag. Their method is to use a "container" in each node to store both the flag and the variable to be reduced.…”

“…A packing function checks whether it is possible to neglect the exponent bits and then use the “fast path”; if not, the “slow path” will be used to reduce the values. For more details, we refer to Reference 16. In our method, we rely on SIMD read/write operations to transport flags and partial reduction values at the same time, which allows us to reduce up to 7 doubles for each thread.…”

“…This gives the advantage that only one barrier is needed to both synchronize the threads and reduce the values. For example, Caballero 15 uses the software combining tree barrier to compute the reduction operation, and the authors in Reference 16 use the Tournament barrier. However, during the thread synchronization, we have to be careful about the communication between threads, especially about the sharing of specific variables.…”

“…This payload is necessary to avoid false sharing during the barrier. An operation reduction combined with a tournament barrier proposed in Reference 16 uses this payload by transporting the reduction values and the synchronization flag on the same cache line. This article was the first to propose handling the payload cache line in barrier synchronization during the reduction operation combined with a barrier.…”