Predictive Analysis of Large-Scale Coupled CFD Simulations with the CPX Mini-App

Abstract

As the complexity of multi-physics simulations increases, there is a need for efficient flow of information between components. Discrete 'coupler' codes can abstract away this process, improving solver interoperability. One such multi-physics problem is modelling the high pressure compressor of turbofan engines, where instances of rotor/stator CFD simulations are coupled. Configuring couplers and allocating resources correctly can be challenging for such problems due to the sliding interfaces between codes. In this research, we present CPX, a mini-coupler designed to model the performance behaviour of a production coupler framework at Rolls-Royce plc., used for coupling rotor/stator simulations. CPX, the first mini-coupler framework of its kind, is combined with a CFD mini-app to predict the run-time and scaling behaviour of large scale coupled CFD simulations. We demonstrate high qualitative and quantitative predictive accuracy with a less than 17% mean error. A performance model is developed to predict the 'optimum' configuration of resources, and is tested to show the high accuracy of these predictions. The model is also used to project the 'optimum' configuration for a 6 Billion cell test case, a problem size representative of current leading-edge production workloads, on a 100,000 core cluster and a 400 GPU cluster. Further testing reveals that the 'optimum' configuration is unstable if not set up correctly, and therefore a trade-off needs to be made with a marginally lessthan-optimal setup to ensure stability. The work illustrates the significant utility of CPX to carry out such rapid design space and run-time setup exploration studies to obtain the best performance from production CFD coupled simulations.

Towards Virtual Certification of Gas Turbine Engines With Performance-Portable Simulations

“…It also simplifies code maintenance and extension. We view coupling as implementing essentially a horizontal separation of concerns [33], where expertise in developing different simulation models for different domains can be leveraged to gain the best results.…”

“…For Rig250, this calculation represents the most performance critical aspect of a CUs operation. Each mesh element in an interface needs to compare itself with every other cell in the other interface, and repeated for all interfaces, both moving and static [33]. Once the "linking" has been found the CUs can transfer the required data from one HS interface to the other usually involving interpolation of data and communication.…”

“…Given the intensity of the interface search routines a further decomposition of work can be implemented by partitioning the interface mesh and allocating separate segments to CUs, thus allowing multiple CUs to work on separate parts of a single interface. This then significantly reduces the search time as discussed in [33]. The JM76 coupler used in this paper makes use of a new parallel binary tree-search algorithm, replacing the previous "brute-force" sequential search from [48].…”

Predictive Analysis of Code Optimisations on Large-Scale Coupled CFD-Combustion Simulations using the CPX Mini-App

“…When coupling between component simulations, codes are tasked with communicating boundary information, where the coupler(s) map values/fields from one simulation to the other, interpolating data and transferring them as required [13]. A key challenge to gain efficient execution is load balancing.…”

“…The idea follows on from the widely used technique in HPC where simplified versions of large applications, called mini-apps, are used to explore co-design, performance and optimum configurations of applications on HPC systems [14], [15]. Building on the coupled multi-row compressor mini-app from [13] with MG-CFD [16] and CPX [13] as the proxy for the density-solver and coupler respectively, a new pressure solver proxy modelled by the SIMPIC [17] Particle-In-Cell (PIC) code is added to create the compressor-combustor-turbine triple-components as a coupled mini-app simulation. A comparison of the full-scale and proxy simulations can be seen in Figure 1, along with a diagram highlighting the related engine components simulated.…”