Using GPUs to improve multigrid solver performance on a cluster

Abstract

This article explores the coupling of coarse and fine-grained parallelism for Finite Element simulations based on efficient parallel multigrid solvers. The focus lies on both system performance and a minimally invasive integration of hardware acceleration into an existing software package, requiring no changes to application code. Because of their excellent price performance ratio, we demonstrate the viability of our approach by using commodity graphics processors (GPUs) as efficient multigrid preconditioners. We address the issue of limited precision on GPUs by applying a mixed precision, iterative refinement technique. Other restrictions are also handled by a close interplay between the GPU and CPU. From a software perspective, we integrate the GPU solvers into the existing MPI-based Finite Element package by implementing the same interfaces as the CPU solvers, so that for the application programmer they are easily interchangeable. Our results show that we do not compromise any software functionality and gain speedups of two and more for large problems. Equipped with this additional option of hardware acceleration we compare different choices in increasing the performance of a conventional, commodity based cluster by increasing the number of nodes, replacement of nodes by a newer technology generation, and adding powerful graphics cards to the existing nodes.

Exploring weak scalability for FEM calculations on a GPU-enhanced cluster

“…Section 2.4), we are not able to perform any meaningful tests for strong scalability as well. Feast has been shown to scale well [2], and we have previously performed limited strong scalability tests on few, but more powerful GPUs [12]. Thus, we can assume that the strong scalability holds true on more GPU nodes as well.…”

“…Here, we only very briefly summarize the ideas and focus on the changes to the Feast kernel while treating the GPUbased multigrid solver as a black box. For a more detailed description of the implementation and a discussion of various tradeoffs, we refer to previous work [12].…”

“…Section 3.6); the data required for a multigrid solver on a large macro just barely fits into the video memory. On better GPUs with more local memory, we can store the matrix data for more than one macro, and we still have enough memory available to software-pipeline the code to reduce delays due to data transfer (see [12] for previous work on few, but more powerful GPUs).…”

“…A 1g 1c configuration is not considered because previous results show that it behaves very similarly to the GPU only (1g 0c) configuration [12]. The total number of unknowns can be derived from R, C and P .…”

“…We estimate 5 W/GB power consumption for the memory modules in DQ. The higher power consumption of newer GPUs is balanced by their speed: NVIDIA's Quadro FX 4500 GPUs consume 95 W each and execute the test problems of this paper (without the stencilbased matrix-vector multiplication) three times faster than the CPUs [12], achieving an efficiency ratio of roughly 3.0/1.3 = 2.3. Consequently, GPUs are also favorable in view of the costs related to operating a cluster, not only in terms of acquisition cost.…”