A Holistic Scalable Implementation Approach of the Lattice Boltzmann Method for CPU/GPU Heterogeneous Clusters

Abstract

Heterogeneous clusters are a widely utilized class of supercomputers assembled from different types of computing devices, for instance CPUs and GPUs, providing a huge computational potential. Programming them in a scalable way exploiting the maximal performance introduces numerous challenges such as optimizations for different computing devices, dealing with multiple levels of parallelism, the application of different programming models, work distribution, and hiding of communication with computation. We utilize the lattice Boltzmann method for fluid flow as a representative of a scientific computing application and develop a holistic implementation for large-scale CPU/GPU heterogeneous clusters. We review and combine a set of best practices and techniques ranging from optimizations for the particular computing devices to the orchestration of tens of thousands of CPU cores and thousands of GPUs. Eventually, we come up with an implementation using all the available computational resources for the lattice Boltzmann method operators. Our approach shows excellent scalability behavior making it future-proof for heterogeneous clusters of the upcoming architectures on the exaFLOPS scale. Parallel efficiencies of more than 90% are achieved leading to 2604.72 GLUPS utilizing 24,576 CPU cores and 2048 GPUs of the CPU/GPU heterogeneous cluster Piz Daint and computing more than 6.8 × 10 9 lattice cells.

Dynamic Load Balancing Techniques for Particulate Flow Simulations

“…In case of larger differences, like e.g. using a GPU instead of a CPU to run the simulation [34,35], this might no longer be the case and the measurements and fits should be redone. A possible improvement to overcome this drawback would be to also add hardware details, like cache sizes, clock frequency, etc., to the estimator and try to come up with a performance model for hardware-aware predictions.…”

Fully parallelized Lattice Boltzmann scheme for fast extraction of biomedical geometry

“…Our method can completely run on GPUs which achieves great performance for biomedical geometry extraction from CT and MRI images. Recently, LBM computations have been implemented on multi-core CPU platforms [35], GPU clusters [36] and heterogeneous CPU/GPU clusters [37]. Our LBM algorithm has similar computational structure and procedure to these methods, while the extra regularization step keeps the parallelism and locality.…”

Performance Limits Study of Stencil Codes on Modern GPGPUs

“…This foreshadowed the trend for 2.5D blocking 1D streaming algorithm [13,24,31], which may be used in conjunction with temporal blocking [20]. The best performance of the applied stencil codes reaches about 30% of the peak theoretical performance [17,18,29].…”