An MPI-CUDA Implementation for Massively Parallel Incompressible Flow Computations on Multi-GPU Clusters

Abstract

Modern graphics processing units (GPUs) with many-core architectures have emerged as general-purpose parallel computing platforms that can accelerate simulation science applications tremendously. While multi-GPU workstations with several TeraFLOPS of peak computing power are available to accelerate computational problems, larger problems require even more resources. Conventional clusters of central processing units (CPU) are now being augmented with multiple GPUs in each compute-node to tackle large problems. The heterogeneous architecture of a multi-GPU cluster with a deep memory hierarchy creates unique challenges in developing scalable and efficient simulation codes. In this study, we pursue mixed MPI-CUDA implementations and investigate three strategies to probe the efficiency and scalability of incompressible flow computations on the Lincoln Tesla cluster at the National Center for Supercomputing Applications (NCSA). We exploit some of the advanced features of MPI and CUDA programming to overlap both GPU data transfer and MPI communications with computations on the GPU. We sustain approximately 2.4 TeraFLOPS on the 64 nodes of the NCSA Lincoln Tesla cluster using 128 GPUs with a total of 30,720 processing elements. Our results demonstrate that multi-GPU clusters can substantially accelerate computational fluid dynamics (CFD) simulations.

Two-fluid compressible simulations on GPU cluster

“…This allows a computation on several GPUs simultaneously. It will allow us to consider finer meshes and also reduce the computation time [1,11,13].…”

GPU-accelerated Large-Eddy Simulation of Turbulent Channel Flows

“…LES capability was integrated into the MPI-CUDA 3D incompressible flow solver developed by Thibault et al 9 and Jacobsen et al 10,11 In their implementation, communication is overlapped with the calculations to increase performance. In this section, we give a brief summary of their work.…”

“…Our present work on LES on a GPU cluster builds upon early work done by Thibault and Senocak 9 and Jacobsen et al 10,11 where they developed a 3D MPI-CUDA parallel incompressible flow solver, called GIN3D. Governing equations are solved using a projection algorithm 24 on a staggered, Cartesian grid with a full-depth parallel geometric multigrid pressure Poisson solver.…”

“…In the field of computational fluid dynamics (CFD), some of the examples include: Thibault and Senocak 9 who focused on an incompressible flow solver, Jacobsen et al who transformed the work of Thibault and Senocak into a Navier-Stokes solver for GPU clusters 10 with a full-depth amalgamated parallel 3D geometric multigrid method 11 21 which gives flexibility to programmers by allowing CUDA applications to be tested on CUDA-enabled GPUs, multicore x86 CPU architectures, or both. CUDA-x86 is the first native CUDA compiler capable of directly creating a binary from CUDA source code for x86 platforms.…”

“…There is a need for GPU clusters and multi-GPU parallel CFD models to study turbulent flows that are common in engineering practice. For the near future, dual-level parallelism that interleaves CUDA with Message Passing Interface (MPI) appear to be an adequate choice to address multi-GPU parallelism 10,15,16 . Jacobsen and Senocak 23 investigated trilevel parallelism using CUDA, MPI, and OpenMP for clusters with multiple GPUs per node.…”