PTG: An Abstraction for Unhindered Parallelism

Abstract

Increased parallelism and use of heterogeneous computing resources is now an established trend in High Performance Computing (HPC), a trend that, looking forward to Exascale, seems bound to intensify. Despite the evolution of hardware over the past decade, the programming paradigm of choice was invariably derived from Coarse Grain Parallelism with explicit data movements. We argue that message passing has remained the de facto standard in HPC because, until now, the ever increasing challenges that application developers had to address to create efficient portable applications remained manageable for expert programmers. Data-flow based programming is an alternative approach with significant potential. In this paper, we discuss the Parameterized Task Graph (PTG) abstraction and present the specialized input language that we use to specify PTGs in our data-flow task-based runtime system, PaRSEC. This language and the corresponding execution model are in contrast with the execution model of explicit message passing as well as the model of alternative task based runtime systems. The Parameterized Task Graph language decouples the expression of the parallelism in the algorithm from the controlflow ordering, load balance, and data distribution. Thus, programs are more adaptable and map more efficiently on challenging hardware, as well as maintain portability across diverse architectures. To support these claims, we discuss the different challenges of HPC programming and how PaR-SEC can address them, and we demonstrate that in today's large scale supercomputers, PaRSEC can significantly outperform state-of-the-art MPI applications and libraries, a trend that will increase with future architectural evolution.

Evaluation of dataflow programming models for electronic structure theory

“…The PTG programming paradigm—which enables explicit dataflow representation of programs—proposes a completely different path from the way parallel applications have been designed and developed up to the present. The PTG decouples the expression of parallelism in the algorithm from the control flow ordering, data distribution, and load balance . Despite the lower startup overhead of implicit dataflow paradigms in terms of development effort (ie, simply submitting tasks in the sequential flow of the original code), the significance of the increased implementation effort from the PTG becomes apparent when comparing the superior performance of the explicit dataflow version of the CC to the implicit dataflow version and to the traditional CC computation.…”

“…The PTG decouples the expression of parallelism in the algorithm from the control flow ordering, data distribution, and load balance. 4 Despite the lower startup overhead of implicit dataflow paradigms in terms of development effort (ie, simply submitting tasks in the sequential flow of the original code), the significance of the increased implementation effort from the PTG becomes apparent when comparing the superior performance of the explicit dataflow version of the CC to the implicit dataflow version and to the traditional CC computation. The PTG version of the CC outperforms the original CC version by a significant margin-to be precise, by a factor of 2.6 on 32 nodes.…”

“…runtime to define the ordering of tasks and can be either implicitly detected by the runtime or explicitly provided by the developer through a specialized notation. This work adopts dataflow programming models to the current state-of-the-art NWChem coupled cluster methods (CC) 1,2 ; where the "explicit" dataflow model takes advantage of the parameterized task graph (PTG) representation [3][4][5] ; and the alternative "implicit" dataflow model uses a task-based representation, where the dataflow is derived by the runtime and the user does not explicitly define the dataflow as they would for the PTG representation. In this paper, we present various implicit dataflow versions of NWChem CC and compare the performance of each runtime.…”

Increasing the degree of parallelism using speculative execution in task-based runtime systems

“…It allows building of graphs using the PTG model, but it also supports various other features such as memory allocation management and parallel containers. PaRSEC (Danalis et al, 2014) is another RS based on the PTF model that had been demonstrated to be effective in various scientific applications. CHARM++ (Kale & Krishnan, 1993) is a C++-based parallel programming system.…”

TaskUniVerse: A Task-Based Unified Interface for Versatile Parallel Execution

“…The StarPU [5] and OmpSs [10] frameworks extend the C compiler and allow the programmer to use compiler directives to define C functions as tasks kernels and describe their data dependencies. The PaRSEC [9,11] framework provides tools and utilities to analyze a program written in a special language that describes tasks and data dependencies and us es a source to source compiler to translate the optimal solution into a C code for compilation. The DuctTeip [22], Chunks and Tasks [17] and also StarPU [2] frameworks provide Application Programming Interface (API) for defining data and tasks to run in a distributed memory environment.…”