Taming the parallel effect zoo

Nested parallelism has proved to be a popular approach for programming the rapidly expanding range of multicore computers. It allows programmers to express parallelism at a high level and relies on a run-time system and a scheduler to deliver efficiency and scalability. As a result, many programming languages and extensions that support nested parallelism have been developed, including in C/C++, Java, Haskell, and ML. Yet, writing efficient and scalable nested parallel programs remains challenging, primarily due to difficult concurrency bugs arising from destructive updates or effects. For decades, researchers have argued that functional programming can simplify writing parallel programs by allowing more control over effects but functional programs continue to underperform in comparison to parallel programs written in lower-level languages. The fundamental difficulty with functional languages is that they have high demand for memory, and this demand only grows with parallelism. In this paper, we identify a memory property, called disentanglement, of nested-parallel programs, and propose memory management techniques for improved efficiency and scalability. Disentanglement allows for (destructive) effects as long as concurrently executing threads do not gain knowledge of the memory objects allocated by each other. We formally define disentanglement by considering an ML-like higher-order language with mutable references and presenting a dynamic semantics for it that enables reasoning about computation graphs of nested parallel programs. Based on this graph semantics, we formalize a classic correctness propertyÐ determinacy race freedomÐand prove that it implies disentanglement. This establishes that disentanglement applies to a relatively broad class of parallel programs. We then propose memory management techniques for nested-parallel programs that take advantage of disentanglement for improved efficiency and scalability. We show that these techniques are practical by extending the MLton compiler for Standard ML to support this form of nested parallelism. Our empirical evaluation shows that our techniques are efficient and scale well. CCS Concepts: • Software and its engineering → Garbage collection; Parallel programming languages; Functional languages; • Theory of computation → Parallel algorithms.

Section: Related Workmentioning

confidence: 96%

Section: Sorting Competitionmentioning

confidence: 99%

Disentanglement in nested-parallel programs

Westrick

Yadav

Fluet

et al. 2019

“…Functional programming languages typically offer substantial control over side effects, usually via powerful type systems [Gifford and Lucassen 1986;Kuper and Newton 2013;Kuper et al 2014a;Launchbury and Peyton Jones 1994;Lucassen and Gifford 1988;Park et al 2008;Peyton Jones and Wadler 1993;Reynolds 1978;Steele 1994;Terauchi and Aiken 2008], which help programmers to avoid race conditions. Notable functional parallel languages include several forms of a Parallel ML language [Acar et al 2015;Fluet et al 2008Raghunathan et al 2016;Westrick et al 2020], the MultiMLton project [Sivaramakrishnan et al 2014;Ziarek et al 2011], the SML# project [Ohori et al 2018], and the work on several forms of Parallel Haskell [Chakravarty et al 2007;Keller et al 2010;Marlow and Jones 2011].…”

Section: Related Workmentioning

confidence: 99%

Provably space-efficient parallel functional programming

Arora

Westrick

Acar

2021

Because of its many desirable properties, such as its ability to control effects and thus potentially disastrous race conditions, functional programming offers a viable approach to programming modern multicore computers. Over the past decade several parallel functional languages, typically based on dialects of ML and Haskell, have been developed. These languages, however, have traditionally underperformed procedural languages (such as C and Java). The primary reason for this is their hunger for memory, which only grows with parallelism, causing traditional memory management techniques to buckle under increased demand for memory. Recent work opened a new angle of attack on this problem by identifying a memory property of determinacy-race-free parallel programs, called disentanglement, which limits the knowledge of concurrent computations about each other’s memory allocations. The work has showed some promise in delivering good time scalability. In this paper, we present provably space-efficient automatic memory management techniques for determinacy-race-free functional parallel programs, allowing both pure and imperative programs where memory may be destructively updated. We prove that for a program with sequential live memory of R * , any P -processor garbage-collected parallel run requires at most O ( R * · P ) memory. We also prove a work bound of O ( W + R * P ) for P -processor executions, accounting also for the cost of garbage collection. To achieve these results, we integrate thread scheduling with memory management. The idea is to coordinate memory allocation and garbage collection with thread scheduling decisions so that each processor can allocate memory without synchronization and independently collect a portion of memory by consulting a collection policy, which we formulate. The collection policy is fully distributed and does not require communicating with other processors. We show that the approach is practical by implementing it as an extension to the MPL compiler for Parallel ML. Our experimental results confirm our theoretical bounds and show that the techniques perform and scale well.

“…Every variable update (łputž) takes the least upper bound of the variable's current and new state with respect to the lattice. The LVish Haskell library [Kuper et al 2014a] implements the LVars programming model and extends it with other deterministic parallel patterns, such as atomically incrementing a counter.…”

Section: Deterministic Parallel Programmingmentioning

confidence: 99%

Concurrency-aware object-oriented programming with roles

Faes

Groß

2018

Object-oriented Programming has been effective in reducing code complexity in sequential programs, but in current practice, concurrent programs still present a number of challenges. We present here a model of object-oriented programming that identifies concurrent tasks and the relationship between objects and tasks, effectively making objects concurrency-aware. This awareness is formalized in a parallel programming model where every object plays a role in every task (e.g., the readonly role). When an object is shared with a new task, it adapts to the new sharing pattern by changing its roles, and therefore its behavior, i.e., the operations that can be performed with this object. This mechanism can be leveraged to prevent interfering accesses from concurrently executing tasks, and therefore makes parallel execution deterministic. To this end, we present a role-based programming language that includes several novel concepts (role transitions, guarding, slicing) to enable practical, object-oriented deterministic parallel programming. We show that this language can be used to safely implement programs with a range of different parallel patterns. The implementations to 8 widely used programming problems achieve substantial parallel speedups and demonstrate that this approach delivers performance roughly on par with manually synchronized implementations. CCS Concepts: • Software and its engineering → Parallel programming languages; Object oriented languages; Concurrent programming languages; Concurrent programming structures; • Theory of computation → Parallel computing models;