A Full-Depth Amalgamated Parallel 3D Geometric Multigrid Solver for GPU Clusters

Abstract

Numerical computations of incompressible flow equations with pressure-based algorithms necessitate the solution of an elliptic Poisson equation, for which multigrid methods are known to be very efficient. In our previous work we presented a dual-level (MPI-CUDA) parallel implementation of the Navier-Stokes equations to simulate buoyancy-driven incompressible fluid flows on GPU clusters with simple iterative methods while focusing on the scalability of the overall solver. In the present study we describe the implementation and performance of a multigrid method to solve the pressure Poisson equation within our MPI-CUDA parallel incompressible flow solver. Various design decisions and algorithmic choices for multigrid methods are explored in light of NVIDIA's recent Fermi architecture. We discuss how unique aspects of an MPI-CUDA implementation for GPU clusters is related to the software choices made to implement the multigrid method. We propose a new coarse grid solution method of embedded multigrid with amalgamation and show that the parallel implementation retains the numerical efficiency of the multigrid method. Performance measurements on the NCSA Lincoln and TACC Longhorn clusters are presented for up to 64 GPUs.

Efficient Two-Level Preconditioned Conjugate Gradient Method on the GPU

“…More recently in [3] also multigrid has been investigated for solving Poisson type problems. In [11] a comparative study is presented between deflation and multigrid.…”

GPU-accelerated Large-Eddy Simulation of Turbulent Channel Flows

“…LES capability was integrated into the MPI-CUDA 3D incompressible flow solver developed by Thibault et al 9 and Jacobsen et al 10,11 In their implementation, communication is overlapped with the calculations to increase performance. In this section, we give a brief summary of their work.…”

“…The pressure Poisson equation (Equation 20) is solved with a full-depth amalgamated geometric multigrid solver 11 . The multigrid method can be separated into four parts: smoothing, restriction, coarse grid solve and prolongation 35,36 .…”

“…Our present work on LES on a GPU cluster builds upon early work done by Thibault and Senocak 9 and Jacobsen et al 10,11 where they developed a 3D MPI-CUDA parallel incompressible flow solver, called GIN3D. Governing equations are solved using a projection algorithm 24 on a staggered, Cartesian grid with a full-depth parallel geometric multigrid pressure Poisson solver.…”

“…In the field of computational fluid dynamics (CFD), some of the examples include: Thibault and Senocak 9 who focused on an incompressible flow solver, Jacobsen et al who transformed the work of Thibault and Senocak into a Navier-Stokes solver for GPU clusters 10 with a full-depth amalgamated parallel 3D geometric multigrid method 11 21 which gives flexibility to programmers by allowing CUDA applications to be tested on CUDA-enabled GPUs, multicore x86 CPU architectures, or both. CUDA-x86 is the first native CUDA compiler capable of directly creating a binary from CUDA source code for x86 platforms.…”