GGAS: Global GPU address spaces for efficient communication in heterogeneous clusters

Abstract

Modern GPUs are powerful high-core-count processors, which are no longer used solely for graphics applications, but are also employed to accelerate computationally intensive general-purpose tasks. For utmost performance, GPUs are distributed throughout the cluster to process parallel programs. In fact, many recent high-performance systems in the TOP500 list are heterogeneous architectures. Despite being highly effective processing units, GPUs on different hosts are incapable of communicating without assistance from a CPU. As a result, communication between distributed GPUs suffers from unnecessary overhead, introduced by switching control flow from GPUs to CPUs and vice versa. Most communication libraries even require intermediate copies from GPU memory to host memory. This overhead in particular penalizes small data movements and synchronization operations, reduces efficiency and limits scalability. In this work we introduce global address spaces to facilitate direct communication between distributed GPUs without CPU involvement. Avoiding context switches and unnecessary copying dramatically reduces communication overhead. We evaluate our approach using a variety of workloads including low-level latency and bandwidth benchmarks, basic synchronization primitives like barriers, and a stencil computation as an example application. We see performance benefits of up to 2× for basic benchmarks and up to 1.67× for stencil computations

Energy-Efficient Collective Reduce and Allreduce Operations on Distributed GPUs

“…For larger messages, GGAS and MPI don't differ strongly, still GGAS is a little bit better. However, this corresponds to our previous results [1] that show that GGAS is performing best for small and medium size data transfers.…”

“…The second one is required after the broadcast operation. However, although GGAS allows fast synchronization between the GPUs [1], every synchronization requires additional data transfers and adds overhead to the application. Especially for small sizes, this synchronization overhead may surpass the data transfer latency.…”

“…1 the simplified user view of a GGAS GPU cluster is shown. The distributed device 1 To avoid confusions with the similar names of the global device memory and the global address space, we will use the term device memory for the global device memory of a single GPU. The term global address space is used for the memory region composed by shared device memories of multiple, distributed GPUs at cluster level.…”

“…However, for such a communicationcentric architecture, minimal costs for communication and synchronization are mandatory to maintain scalability, performance and energy efficiency. In previous work [1] we have introduced GPU Global Address Spaces (GGAS) as a communication model that is perfectly in-line with the GPUs thread-collective execution model. GGAS not only allows increasing performance for a variety of workloads, it also avoids hybrid programming models like MPI+CUDA and complete bypasses the CPU for inter-GPU communication tasks.…”

CUDA-Aware OpenSHMEM: Extensions and Designs for High Performance OpenSHMEM on GPU Clusters

“…Similarly MIC-RO [13] enables sharing and using multiple Intel Many Integrated Core (MIC), cards across nodes. The work in [14] proposed concepts that allow an HCA to access GPU memory similar to GDR. However, their concepts require specific hardware and cannot be applied to production ready HPC systems.…”