Deterministic parallel random-number generation for dynamic-multithreading platforms

Abstract

Existing concurrency platforms for dynamic multithreading do not provide repeatable parallel random-number generators. This paper proposes that a mechanism called pedigrees be built into the runtime system to enable efficient deterministic parallel randomnumber generation. Experiments with the open-source MIT Cilk runtime system show that the overhead for maintaining pedigrees is negligible. Specifically, on a suite of 10 benchmarks, the relative overhead of Cilk with pedigrees to the original Cilk has a geometric mean of less than 1%.We persuaded Intel to modify its commercial C/C++ compiler, which provides the Cilk Plus concurrency platform, to include pedigrees, and we built a library implementation of a deterministic parallel random-number generator called DOTMIX that compresses the pedigree and then "RC6-mixes" the result. The statistical quality of DOTMIX is comparable to that of the popular Mersenne twister, but somewhat slower than a nondeterministic parallel version of this efficient and high-quality serial random-number generator. The cost of calling DOTMIX depends on the "spawn depth" of the invocation. For a naive Fibonacci calculation with n = 40 that calls DOTMIX in every node of the computation, this "price of determinism" is about a factor of 2.3 in running time over the nondeterministic Mersenne twister, but for more realistic applications with less intense use of random numbers -such as a maximalindependent-set algorithm, a practical samplesort program, and a Monte Carlo discrete-hedging application from QuantLib -the observed "price" was at most 21%, and sometimes much less. Moreover, even if overheads were several times greater, applications using DOTMIX should be amply fast for debugging purposes, which is a major reason for desiring repeatability.

GraphGrind

“…The NUMA extension supports non-commuting reductions [6] and pedigrees [16]. Both constructs depend on the execution order of function calls, which the helper function disrupts.…”

Designing Bit-Reproducible Portable High-Performance Applications

“…Figure 1 shows the performance loss of the reproducible implementation of transcendental functions with respect to the standard library installed on different systems, i.e., Glibc for the CPU and the CUDA toolkit for GPUs. The loss of performance is defined as the ratio of the time required by the deterministic implementation to the time required by the standard one on the corresponding platform to perform the task of evaluating the function on 2 22 input values. Figure 2 shows the geometric mean of the performance loss of all implemented functions for every architecture.…”

“…Other studies on non-determinism caused by parallelism have been performed by Bergan et al [20], Bocchino et al [21], Leiserson et al [22], Olszewski et al [23].…”

Internally deterministic parallel algorithms can be fast

“…Not only must the output of the program be deterministic, but all intermediate values returned from operations must also be deterministic. We note that this does not preclude the use of pseudorandom numbers, where one can use, for example, the approach of Leiserson et al [33] to generate deterministic pseudorandom numbers in parallel from a single seed, which can be part of the input.…”