Parallel Reproducible Summation

Abstract

Reproducibility, i.e. getting bitwise identical floating point results from multiple runs of the same program, is a property that many users depend on either for debugging or correctness checking in many codes [10]. However, the combination of dynamic scheduling of parallel computing resources, and floating point nonassociativity, makes attaining reproducibility a challenge even for simple reduction operations like computing the sum of a vector of numbers in parallel. We propose a technique for floating point summation that is reproducible independent of the order of summation. Our technique uses Rump's algorithm for error-free vector transformation [7], and is much more efficient than using (possibly very) high precision arithmetic. Our algorithm reproducibly computes highly accurate results with an absolute error bound of n Á 2 À28 Á macheps Á max i jv i j at a cost of 7n FLOPs and a small constant amount of extra memory usage. Higher accuracies are also possible by increasing the number of error-free transformations. As long as all operations are performed in to-nearest rounding mode, results computed by the proposed algorithms are reproducible for any run on any platform. In particular, our algorithm requires the minimum number of reductions, i.e. one reduction of an array of six double precision floating point numbers per sum, and hence is well suited for massively parallel environments.

Numerical reproducibility for the parallel reduction on multi- and many-core architectures

“…As Section 4 shows, our algorithm is faster in the bandwidth-constrained scenarios with moderate dynamic ranges. Demmel and Nguyen have also improved the previous results [28,22] by using one single reduction step among nodes. Such an improvement yielded roughly 20 % overhead on 1024 processors compared to the Intel MKL dasum(), but it shows roughly 3.4 times slowdown on 32 processors.…”

“…The one reduction reproducible summation, Alg. 6 Sequential Reproducible Summation [22], (referred as ReproBLAS) from the ReproBLAS library 6 ; 5. The single-sweep reduction [23] with two and three levels (cited as bitrep2 and bitrep3, accordingly) from the bitrep library 7 .…”

Reproducible, Accurately Rounded and Efficient BLAS

“…We have implemented an OpenMP parallel version of this algorithm since ReproBLAS offers only an MPI parallel version. We derive reproducible version of dot, nrm2, asum and gemv by replacing all non-associative accumulations by the algorithm OneReduction [6]. These versions are denoted OneReductionDot, OneReductionAsum, OneReductionN rm2 and OneReductionGemv.…”

“…Therefore numerical results do not depend anymore on hardware configuration. The performance of these latter is improved with the algorithm OneReduction [6] by relying on indexed floating-point numbers [5] and requiring a single reduction operation to reduce the communication cost on distributed memory parallel platforms. However, those solutions do not improve accuracy.…”

Reproducible floating-point atomic addition in data-parallel environment

“…Strong numerical reproducibility can be further subdivided in two classes of algorithms. Those which are producing correctly rounded results such as the one based on long accumulators [7], and others which provide reproducible results without any guaranty on accuracy [8].…”