Efficient and Scalable Initialization of Partitioned Coupled Simulations with preCICE

Abstract

preCICE is an open-source library, that provides comprehensive functionality to couple independent parallelized solver codes to establish a partitioned multi-physics multi-code simulation environment. For data communication between the respective executables at runtime, it implements a peer-to-peer concept, which renders the computational cost of the coupling per time step negligible compared to the typical run time of the coupled codes. To initialize the peer-to-peer coupling, the mesh partitions of the respective solvers need to be compared to determine the point-to-point communication channels between the processes of both codes. This initialization effort can become a limiting factor, if we either reach memory limits or if we have to re-initialize communication relations in every time step. In this contribution, we remove two remaining bottlenecks: (i) We base the neighborhood search between mesh entities of two solvers on a tree data structure to avoid quadratic complexity, and (ii) we replace the sequential gather-scatter comparison of both mesh partitions by a two-level approach that first compares bounding boxes around mesh partitions in a sequential manner, subsequently establishes pairwise communication between processes of the two solvers, and finally compares mesh partitions between connected processes in parallel. We show, that the two-level initialization method is fives times faster than the old one-level scheme on 24,567 CPU-cores using a mesh with 628,898 vertices. In addition, the two-level scheme is able to handle much larger computational meshes, since the central mesh communication of the one-level scheme is replaced with a fully point-to-point mesh communication scheme.

preCICE v2: A sustainable and user-friendly coupling library

“…For a more classical description of preCICE including classical validation with benchmarks, we refer the reader to the first reference paper 18 . Performance-focused publications 17,21 also demonstrate recent performance and scalability improvements. The contributions of this paper enable new scientific insights in the research fields of our users, but also provide new experiences in scientific software engineering.…”

“…To this end, bounding boxes around mesh partitions are compared in a first step, leading to preliminary communication channels. In a second step, actual mesh data is compared in a fully-parallel fashion 21 .…”

“…Several alternative versions/forks exist 19. OpenFOAM adapter documentation: https://www.precice.org/adapter-openfoam-overview.html20 OpenFOAM Journal: https://journal.openfoam.com/21 OpenFOAM adapter on GitHub: https://github.com/precice/openfoam-adapter, GPLv322 OpenFOAM User Guide -chtMultiRegionFoam: https://www.openfoam.com/documentation/guides/latest/doc/guide-applications-solvers-heat-transfer-chtMultiRe-gionFoam.html.…”

OpenFOAM-preCICE: Coupling OpenFOAM with External Solvers for Multi-Physics Simulations

“…We can interpret these results from different angles, but we focus here on the performance of the adapter implementation. The performance of the preCICE library and of the coupling methods involved has extensively been documented in the literature [5,8,46,52].…”

preCICE v2: A sustainable and user-friendly coupling library

“…When RBF data mapping becomes too expensive, nearest-projection mapping is a good alternative except for very large out-of-memory cases. Scalability results for the mapping computation were recently published in 21 . In future work, we aim for a more in-depth analysis of mapping variants with further geometries.…”