Language support for dynamic, hierarchical data partitioning

Abstract

Applications written for distributed-memory parallel architectures must partition their data to enable parallel execution. As memory hierarchies become deeper, it is increasingly necessary that the data partitioning also be hierarchical to match. Current language proposals perform this hierarchical partitioning statically, which excludes many important applications where the appropriate partitioning is itself data dependent and so must be computed dynamically. We describe Legion, a region-based programming system, where each region may be partitioned into subregions. Partitions are computed dynamically and are fully programmable. The division of data need not be disjoint and subregions of a region may overlap, or alias one another. Computations use regions with certain privileges (e.g., expressing that a computation uses a region read-only) and data coherence (e.g., expressing that the computation need only be atomic with respect to other operations on the region), which can be controlled on a per-region (or subregion) basis.We present the novel aspects of the Legion design, in particular the combination of static and dynamic checks used to enforce soundness. We give an extended example illustrating how Legion can express computations with dynamically determined relationships between computations and data partitions. We prove the soundness of Legion's type system, and show Legion type checking improves performance by up to 71% by eliding provably safe memory checks. In particular, we show that the dynamic checks to detect aliasing at runtime at the region granularity have negligible overhead. We report results for three real-world applications running on distributed memory machines, achieving up to 62.5X speedup on 96 GPUs on the Keeneland supercomputer.

A Case for Transforming Parallel Runtimes Into Operating System Kernels

“…The Legion runtime system is designed to provide applications with a parallel programming model that maps well to heterogeneous architectures [1,22]. Whether the application runs on a single node or across nodes-even with GPUs-the Legion runtime can manage the underlying resources so that the application does not have to.…”

“…Some modern runtimes, such as the Legion runtime [1,22] we consider in this paper, already address this challenge by creating abstractions that programmers or the compilers of high-level languages can target, abstractions that mirror the machine in portable ways. Very high-level parallel languages can let us further decouple the expression of parallelism from its implementation.…”

Structure Slicing: Extending Logical Regions with Fields

“…A full treatment of non-interference in original Legion can be found in [25], which derives a sound approximation of the non-interference test that is efficient enough to be performed at run time. It also shows how the functional nature of Legion tasks with effects on logical regions enables a hierarchical scheduling algorithm and permits distributed non-interference tests to be performed on different nodes without communication, which is crucial for scaling Legion to thousands of nodes.…”

“…Following the methodology of [25], we first define a precise non-interference test based on the actual execution of two tasks and then show a sound approximation that can be efficiently computed using region field privileges. In [25], a memory operation is a triple (l, op, v) where l is a memory location, op is the operation performed on l (read, write, or reduce o with a particular reduction operator o), and v is the value (the value read, written or folded into l using the named reduction operator). Non-interference of two memory operations 1 and 2 is then defined as follows:…”

“…To aid programmers in managing program data, we previously introduced Legion, a data-centric programming model based on logical regions [6,25,26]. Logical regions are typed collections that can be recursively partitioned into logical subregions, thereby allowing applications to name the subsets of data used by different sub-computations.…”