Safe nondeterminism in a deterministic-by-default parallel language

Abstract

A number of deterministic parallel programming models with strong safety guarantees are emerging, but similar support for nondeterministic algorithms, such as branch and bound search, remains an open question. We present a language together with a type and effect system that supports nondeterministic computations with a deterministic-by-default guarantee: nondeterminism must be explicitly requested via special parallel constructs (marked nd), and any deterministic construct that does not execute any nd construct has deterministic input-output behavior. Moreover, deterministic parallel constructs are always equivalent to a sequential composition of their constituent tasks, even if they enclose, or are enclosed by, nd constructs. Finally, in the execution of nd constructs, interference may occur only between pairs of accesses guarded by atomic statements, so there are no data races, either between atomic statements and unguarded accesses (strong isolation) or between pairs of unguarded accesses (stronger than strong isolation alone). We enforce the guarantees at compile time with modular checking using novel extensions to a previously described effect system. Our effect system extensions also enable the compiler to remove unnecessary transactional synchronization. We provide a static semantics, dynamic semantics, and a complete proof of soundness for the language, both with and without the barrier removal feature. An experimental evaluation shows that our language can achieve good scalability for realistic parallel algorithms, and that the barrier removal techniques provide significant performance gains.

Language support for dynamic, hierarchical data partitioning

“…Dynamic non-interference of memory operations can only be determined after evaluation of an expression is completed, and only at great expense, as illustrated by work on transactional memory [13]. At the other extreme are systems like Jade [17] and DPJ [5] that check non-interference statically, but must disallow aliasing to do so. In contrast, Legion can verify non-interference of privileges at runtime, which is much simpler and more efficient than checking non-interference of dynamic memory traces.…”

Isolation for nested task parallelism

“…However, those constructs (or the constructs offered by DPJ [6]) are not applicable to the nondeterministic, irregular parallel programs supported by Otello.…”

A programming model for deterministic task parallelism

“…Deterministic Parallel Java [2] uses a static type and effect system to enforce memory isolation among tasks, and uses regions to reason about memory effects and infer task memory footprints. However, parallelism is restricted inside cobegin statements that implicitly wait for all sub-tasks before returning to the parent task, similarly to Cilk.…”

“…1 To overcome this, manufacturers relax the coherent shared memory requirements, moving towards NUMA and non-cache coherent systems. 2 In the past, non-shared memory computers were mostly used in scientific and high-performance applications. To fully take advantage of cluster supercomputers, HPC programmers use explicit message-passing between parallel processes running on separate processors.…”

“…Obviously, the abstraction of shared memory is not as costly in a multicore pro- 1 Spec benchmarks show that memory bus contention limits the performance of the 8-core Xeon processor. 2 The 12-core Opteron processor is NUMA, and gives better per-core scaling than Xeon. The Cell and Intel SCC processors do not have coherent shared memory among cores.…”