Unsteady Turbulent Simulations on a Cluster of Graphics Processors

Abstract

This paper describes the GPU accelerated MBFLO2 multi-block turbulent flow solver completely in double precision using CUDA and the latest generation of GPU processors. On a cluster of 8 Tesla C2050 "Fermi" GPUs and Intel Xeon X5550 "Nehalem" quad-core CPUs, we achieve 9x speedup over the parallel CPU solver or 70x speedup over the serial solver. High performance is obtained by optimizing the data layout on the GPU, optimizing data transfers and using asynchronous memory copies to overlap GPU execution with communications. We test the solver on a turbulent flat plate and an unsteady turbulent cylinder with 3.2 million grid points. We confirm the GPU results are in agreement with turbulent flow theory. We discuss the GPU optimization techniques used to reach this level of performance. Nomenclature E = total energy g = acceleration due to gravity H = total enthalpy h = static enthalpy I = rothalpy k = turbulent kinetic energy p = pressure Pr = Prandtl number Pr t = turbulent Prandtl number R = radius from specified axis of rotation S ij = mean strain-rate tensor u = axial velocity component û = internal energy v = tangential velocity component V = velocity magnitude z = elevation = coefficient of viscosity = turbulent coefficient of viscosity = turbulent dissipation rate divided by turbulent kinetic energy = rotational velocity about specified axis of rotation (rads/s) ij = shear stress tensor = density

Recent progress and challenges in exploiting graphics processors in computational fluid dynamics

“…Phillips et al [86] developed one of the first GPU solvers capable of simulating turbulence using the k-ω model, extending on the group's previous work porting portions of the existing MBFLO solver to the GPU [85]. In addition, their new solver was capable of running on a cluster of multiple CPU/GPU nodes, using a domain decomposition technique to give each node responsibility for a block of the overall domain.…”

“…The CPU only drove the simulation and passed information between the blocks of the domain, using MPI to transfer information between independent cluster nodes. Phillips et al [86] also improved performance by implementing a novel asynchronous memory transfer using CUDA streams; in their previous work, the GPU remained idle while the CPU transferred memory between different blocks (i.e., subdomains). Here, each block was further divided in half such that the GPU could continue to perform calculations on one half while the CPU transferred memory associated with the other half of the block; this improved performance up to 40%.…”

“…Here, each block was further divided in half such that the GPU could continue to perform calculations on one half while the CPU transferred memory associated with the other half of the block; this improved performance up to 40%. Phillips et al [86] tested their code using a simulation of unsteady turbulent flow over a cylinder, finding that a cluster of eight GPUs performed about nine times faster than an equivalent parallel code running on eight quad-core CPUs.…”

“…DeLeon et al [29] parallelized their solver to run on a cluster of multiple GPUs using MPI, such that the overall code contained two levels of parallelism. The overhead of communication between blocks was minimized by using the same asynchronous memory transfer as Phillips et al [86]. They did not compare the performance of their LES GPU solver to an equivalent CPU version, but simulations of turbulent channel flow with approximately 9.4 million grid cells took 45 hours to complete running on a cluster with eight total GPUs.…”

Discretized peridynamics for brittle and ductile solids

“…Manavski et al [43] used GPUs as an accelerator for Smith-Waterman sequence alignment. Phillips et al [44] implemented a multi-block turbulent flow solver in GPU processors.…”