Improving the scalability of the ocean barotropic solver in the community earth system model

Abstract

High-resolution climate simulations are increasingly in demand and require tremendous computing resources. In the Community Earth System Model (CESM), the Parallel Ocean Model (POP) is computationally expensive for high-resolution grids (e.g., 0.1 •) and is frequently the least scalable component of CESM for certain production simulations. In particular, the modified Preconditioned Conjugate Gradient (PCG), used to solve the elliptic system of equations in the barotropic mode, scales poorly at the high core counts, which is problematic for high-resolution simulations. In this work, we demonstrate that the communication costs in the barotropic solver occupy an increasing portion of the total POP execution time as core counts are increased. To mitigate this problem, we implement a preconditioned Chebyshev-type iterative method in POP (called P-CSI), which requires far fewer global reductions than PCG. We also develop an effective block preconditioner based on the Error Vector Propagation Method to attain a competitive convergence rate for P-CSI. We demonstrate that the improved scalability of P-CSI results in a 5.2x speedup of the barotropic mode in high-resolution POP on 16,875 cores, which yields a 1.7x speedup of the overall POP simulation. Further, we ensure that the new solver produces an ocean climate consistent with the original one via an ensemble-based statistical method.

Evaluating statistical consistency in the ocean model component of the Community Earth System Model (pyCECT v2.0)

“…For example, POP-RMSE was unable to quantify the small climate state changes due to recent linear solver modifications in CESM-POP in Hu et al (2015). In Hu et al (2015), the authors replaced the default preconditioned conjugate gradient (PCG) solver for the barotropic mode of POP by an alternative preconditioned Chebyshev-type iterative (P-CSI) solver to enhance the computational performance. While the P-CSI solver had considerably lower communication costs for high-resolution simulations than PCG, showing that the use of an alternative solver did not negatively impact the ocean simulation results was critical for acceptance.…”

“…While the P-CSI solver had considerably lower communication costs for high-resolution simulations than PCG, showing that the use of an alternative solver did not negatively impact the ocean simulation results was critical for acceptance. Therefore, in Hu et al (2015), to gauge the effectiveness of the POP-RMSE test in detecting solver differences over time, monthly data were collected for 36 months from a CESM-POP 1 • resolution case with multiple convergence tolerances between 10 −10 and 10 −16 . Figure 1 displays the RMSE between the strictest case (10 −16 ) and the other tolerances listed in the figure's legend for the temperature field.…”

“…Despite the range in convergence tolerances used, Fig. 1 gives scant evidence of any solver-induced error (Hu et al, 2015).…”

“…Since the RMSE approach did not elucidate differences between convergence tolerances, Hu et al (2015) adapted the ensemble approach that was initially described in the context of atmospheric data compression in Baker et al (2014) Figure 2. Monthly root mean square Z-score (RMSZ) of temperature with respect to an ensemble (denoted by yellow) for experiments with barotropic different convergence tolerances.…”

“…The CESM-POP solves the threedimensional (3-D) primitive equations for ocean dynamics with hydrostatic and Boussinesq approximations, representing ocean processes across a broad range of spatial and temporal scales. Much new development in CESM-POP is aimed A. H. Baker et al: Evaluating statistical consistency at reducing computational costs (e.g., Hu et al, 2015), but ongoing development of any type in a simulation code requires software quality assurance to ensure that no errors are introduced. The need for some sort of quality assurance to maintain confidence in the science results is particularly critical for climate models, whose simulation output may influence policy decisions with broad societal impact (Carson, 2002;Easterbrook et al, 2011).…”