Radiation modeling using the Uintah heterogeneous CPU/GPU runtime system

Abstract

The Uintah Computational Framework was developed to provide an environment for solving fluid-structure interaction problems on structured adaptive grids on large-scale, long-running, data-intensive problems. Uintah uses a combination of fluid-flow solvers and particle-based methods for solids, together with a novel asynchronous task-based approach with fully automated load balancing. Uintah demonstrates excellent weak and strong scalability at full machine capacity on XSEDE resources such as Ranger and Kraken, and through the use of a hybrid memory approach based on a combination of MPI and Pthreads, Uintah now runs on up to 262k cores on the DOE Jaguar system. In order to extend Uintah to heterogeneous systems, with ever-increasing CPU core counts and additional onnode GPUs, a new dynamic CPU-GPU task scheduler is designed and evaluated in this study. This new scheduler enables Uintah to fully exploit these architectures with support for asynchronous, outof-order scheduling of both CPU and GPU computational tasks. A new runtime system has also been implemented with an added multi-stage queuing architecture for efficient scheduling of CPU and GPU tasks. This new runtime system automatically handles the details of asynchronous memory copies to and from the GPU and introduces a novel method of pre-fetching and preparing GPU memory prior to GPU task execution. In this study this new design is examined in the context of a developing, hierarchical GPUbased ray tracing radiation transport model that provides Uintah with additional capabilities for heat transfer and electromagnetic wave propagation. The capabilities of this new scheduler design are tested by running at large scale on the modern heterogeneous systems, Keeneland and TitanDev, with up to 360 and 960 GPUs respectively. On these systems, we demonstrate significant speedups per GPU against a standard CPU core for our radiation problem.

Parallel Block Structured Adaptive Mesh Refinement on Graphics Processing Units

“…The data transfer is performed concurrently with other computation, so the imapact is minimised, and the authors note that data transfer time typcially only takes 30% of the application runtime. The Uintah framework from the University of Utah is an AMR framework that supports GPUs [11,14]. The focus in Uintah is on heterogeneous platforms, and as with GAMER, solution data must be copied between the CPU and GPU memory as required by the numerical kernels.…”

The uintah framework: a unified heterogeneous task scheduling and runtime system

“…Experimental results [1] on typical fluid AMR simulations showed 50% to 90% savings on memory usage. This new multi-threaded MPI scheduler enabled Uintah to scale up to [5] 196K cores on the DoE Jaguar XT5 system and became the basis for the heterogeneous multi-threaded MPI scheduler [5] which allowed Uintah to dispatch tasks to GPUs as well as CPU cores on a node.…”

“…Multi-threaded CPU-GPU Scheduler (Master-Slave Model) [5] In the same fashion that Uintah insulates the application developer from the parallelism its infrastructure provides via the multi-threaded CPU scheduler, the hybrid CPU-GPU version also hides and carefully manages details related to GPU memory allocation and transfer. Associated with each Uintah task is a C++ method which is used to perform the actual computation.…”

“…Experimental results on challenging simulations using Adaptive Mesh Refinement (AMR) showed a 50% to 90% savings on memory usage by using the multi-threaded scheduler. This scheduler was then extended to heterogeneous systems [5], able to dispatch work to both CPU cores and GPUs on-node. This approach still did not result in the scalable execution of the target fluid-structure problem beyond 200K cores however.…”

Scalable Heterogeneous CPU-GPU Computations for Unstructured Tetrahedral Meshes

“…For structured meshes, heterogeneous CPU-GPU computation has been studied in several publications. [2][3][4][5] However, the unstructured nature of our problem poses significant additional challenges with respect to partitioning, communication, and load balancing.…”