GPU-aware Communication with UCX in Parallel Programming Models: Charm++, MPI, and Python

Abstract

As an increasing number of leadership-class systems embrace GPU accelerators in the race towards exascale, efficient communication of GPU data is becoming one of the most critical components of high-performance computing. For developers of parallel programming models, implementing support for GPUaware communication using native APIs for GPUs such as CUDA can be a daunting task as it requires considerable effort with little guarantee of performance. In this work, we demonstrate the capability of the Unified Communication X (UCX) framework to compose a GPU-aware communication layer that serves multiple parallel programming models developed out of the Charm++ ecosystem, including MPI and Python: Charm++, Adaptive MPI (AMPI), and Charm4py. We demonstrate the performance impact of our designs with microbenchmarks adapted from the OSU benchmark suite, obtaining improvements in latency of up to 10.2x, 11.7x, and 17.4x in Charm++, AMPI, and Charm4py, respectively. We also observe increases in bandwidth of up to 9.6x in Charm++, 10x in AMPI, and 10.5x in Charm4py. We show the potential impact of our designs on real-world applications by evaluating weak and strong scaling performance of a proxy application that performs the Jacobi iterative method, improving the communication performance by up to 12.4x in Charm++, 12.8x in AMPI, and 19.7x in Charm4py.

Performance Evaluation of Python Parallel Programming Models: Charm4Py and mpi4py

“…Charm4Py [15] is a parallel programming model built on top of Charm++. Charm4Py features the message-driven scheduling of Charm++ [16], and has support for many Charm++ features such as dynamic load balancing, GPU-direct communication [17], overdecomposition, and sections. Following the programming model of Charm++, Charm4Py programs consist of one or more chares on each PE in the computation.…”

“…In addition to optimizations for host-resident data, Charm4Py and mpi4py are capable of inter-process communication consisting of GPU-resident data without first staging data on the host. Charm4Py uses the underlying UCX capabalities of Charm++ [17], and mpi4py utilizes CUDA-aware MPI implementations.…”