Weighted Decomposition in High-Performance Lattice-Boltzmann Simulations: Are Some Lattice Sites More Equal than Others?

Abstract

Obtaining a good load balance is a significant challenge in scaling up lattice-Boltzmann simulations of realistic sparse problems to the exascale. Here we analyze the effect of weighted decomposition on the performance of the HemeLB lattice-Boltzmann simulation environment, when applied to sparse domains. Prior to domain decomposition, we assign wall and in/outlet sites with increased weights which reflect their increased computational cost. We combine our weighted decomposition with a second optimization, which is to sort the lattice sites according to a space filling curve. We tested these strategies on a sparse bifurcation and very sparse aneurysm geometry, and find that using weights reduces calculation load imbalance by up to 85%, although the overall communication overhead is higher than some of our runs.

GPU Acceleration of the HemeLB Code for Lattice Boltzmann Simulations in Sparse Complex Geometries

“…In other words, HemeLB provides consistent performance for geometries of varying sparsity. This is an interesting result given that previous performance tests of HemeLB by Groen et al [37] reported a notable difference in peak performance for geometries with varying sparsity. Those reports alluded to the possibility that the optimal tradeoff between computation and communication load-imbalance can vary substantially for different HPC systems.…”

“…They reported, in 2013, near-linear scaling up to 32,768 cores. A performance of 153 billion site updates per second (SUP/s) using 49,152 cores on the ARCHER supercomputer was reported by the same authors elsewhere [37].…”

“…There have also been efforts dedicated to weighted decomposition schemes that account for the different computational costs of different site types (e.g. bulk sites, boundary sites, or inlet/outlet sites) in order to improve workload balance [37]. Based on the domain decomposition, each process has a twodimensional array of fluid distributions for its domain.…”

“…The GPU implementation currently uses the existing infrastructure for boundary exchanges in HemeLB with very few modifications, so the integration of GPU memory transfers with MPI communication are not fully optimized. It is estimated that the throughput of 153 billion SUP/s, achieved by Groen et al [37] using nearly 50,000 cores, would require hundreds to thousands of GPUs. Completing the aforementioned integration will be a critical step in bringing HemeLB-GPU to this scale, which would substantially reduce the computational requirements and energy consumption of scientific simulations performed with HemeLB.…”

An automated multiscale ensemble simulation approach for vascular blood flow

“…It is a MPI parallelised C++ code with world-class scalability for sparse geometries. It can efficiently model flows in sparse cerebral arteries using up to 32,768 cores [22,23] and utilises a weighted domain decomposition approach to minimize the overhead introduced by compute-intensive boundary and in-/outflow conditions [8]. HemeLB allows users to obtain key flow properties such as velocity, pressure and wall shear at predefined intervals of time, using a property-extraction framework.…”