Reproducible, Accurately Rounded and Efficient BLAS

Abstract

Abstract. Numerical reproducibility failures rise in parallel computation because floating-point summation is non-associative. Massively parallel and optimized executions dynamically modify the floating-point operation order. Hence, numerical results may change from one run to another. We propose to ensure reproducibility by extending as far as possible the IEEE-754 correct rounding property to larger operation sequences. We introduce our RARE-BLAS (Reproducible, Accurately Rounded and Efficient BLAS) that benefits from recent accurate and efficient summation algorithms. Solutions for level 1 (asum, dot and nrm2) and level 2 (gemv) routines are presented. Their performance is studied compared to the Intel MKL library and other existing reproducible algorithms. For both shared and distributed memory parallel systems, we exhibit an extra-cost of 2× in the worst case scenario, which is satisfying for a wide range of applications. For Intel Xeon Phi accelerator a larger extra-cost (4× to 6×) is observed, which is still helpful at least for debugging and validation steps.

Parallel Experiments with RARE-BLAS

“…2) Dot Product: Our implementation of parallel reproducible dot and nrm2 is presented in [9]. For parallel dot product, we distinguish three steps.…”

“…Note that the local result is not rounded, the dot product is transformed to a sum of non-overlapping floating-point numbers. The process is done in different ways depending on the vector size (more details in [9]). (2) Afterwards, local results of transformation are gathered by master thread.…”

“…All previous transformations do not generate any error. (3) Finally the master thread uses iF astSum [12] to calculate a correctly rounded sum of the gathered data (See Figure 1 in [9]).…”

“…So parallel algorithms for correctly rounded dot and asum and for a faithfully rounded nrm2 have been designed [7]. This approach has been extended to the matrix-vector multiplication from the level 2 BLAS in [9]. First implementation of level 1 BLAS exhibits interesting performance with 2× extra-cost in the worst case scenario on shared memory parallel systems [7].…”

“…First implementation of level 1 BLAS exhibits interesting performance with 2× extra-cost in the worst case scenario on shared memory parallel systems [7]. Other implementations for distributed memory computing systems and accelerators (Intel Xeon Phi) are introduced in [9].…”