P-CSI v1.0, an accelerated barotropic solver for the high-resolution ocean model component in the Community Earth System Model v2.0

Abstract

Abstract. In the Community Earth System Model (CESM), the ocean model is computationally expensive for highresolution grids and is often the least scalable component for high-resolution production experiments. The major bottleneck is that the barotropic solver scales poorly at high core counts. We design a new barotropic solver to accelerate the high-resolution ocean simulation. The novel solver adopts a Chebyshev-type iterative method to reduce the global communication cost in conjunction with an effective block preconditioner to further reduce the iterations. The algorithm and its computational complexity are theoretically analyzed and compared with other existing methods. We confirm the significant reduction of the global communication time with a competitive convergence rate using a series of idealized tests. Numerical experiments using the CESM 0.1 • global ocean model show that the proposed approach results in a factor of 1.7 speed-up over the original method with no loss of accuracy, achieving 10.5 simulated years per wall-clock day on 16 875 cores.

The Community Earth System Model Version 2 (CESM2)

“…The physics advances include a new parameterization for mixing effects in estuaries, which improves the representation of the exchange of freshwater between the terrestrial and marine branches of the hydrologic cycle (Sun et al, 2019); increased mesoscale eddy (isopycnal) diffusivities at depth to improve the representation of passive tracers; use of prognostic chlorophyll for shortwave absorption; use of salinity-dependent freezing-point together with the sea-ice model (Assur, 1958); and a new Langmuir mixing parameterization in conjunction with the new wave model component (Li et al, 2016; also see below). The numerical improvements include a new iterative solver for the barotropic mode to reduce communication costs, particularly advantageous for high-resolution simulations on large processor counts (Huang et al, 2016); a tracer-conserving time filtering scheme based on an adaption of the Robert-Asselin filter to enable subdiurnal coupling of the ocean model (Asselin, 1972;Robert, 1966;Williams, 2011); and subsequently, use of a 1-hr coupling frequency to explicitly resolve (sub) diurnal and inertial periods. In addition, the K-Profile vertical mixing Parameterization (KPP; Large et al, 1994) is incorporated via the Community ocean Vertical Mixing (CVMix) framework, and the Caspian Sea is treated as a lake in the land model, and thus, it is no longer included in the ocean model as a marginal sea.…”

Scalability and some optimization of the Finite-volumE Sea ice-Ocean Model, Version 2.0 (FESOM2)

“…Alternatively, one may follow the suggestion of Huang et al (2016) and switch to a solver based on a Chebyshev method.…”

“…The method of Huang et al (2016) is based on an algorithm without global sums and allows one to better overlap computation with communication. A caveat here is that unstructured meshes used by FESOM may cause the SSH system matrix to be more poorly conditioned than in Huang et al (2016), since mesh elements can differ by a factor of 100 or greater. It remains to be seen whether this method is able to compete with the split-explicit approach.…”

“…However, existing models are still rather slow and require many months of wallclock time for century-scale climate simulations, making prohibitive to use these configurations in many important applications. This cores with a new solver for the external mode (Huang et al (2016)). Throughputs higher than these can still be reached, but would imply a very inefficient use of computational power.…”

Optimizing High-Resolution Community Earth System Model on a Heterogeneous Many-Core Supercomputing Platform (CESM-HR_sw1.0)

“…Notable developments in the CESM1.3 progression from beta17_sehires20 to the current beta17_sehires38 version include two modifications for the POP2 ocean model. The first change involved back-porting a new iterative solver for the barotropic mode from CESM1.2 to CESM1.3 to reduce communication costs, particularly beneficial for high-resolution simulations on large processor counts (e.g., Hu et al, 2015;Huang at al., 2016). The second modification was a change of the ocean coupling frequency from one hour to 30 minutes to alleviate any potential coupling instabilities that may arise with longer coupling frequencies between the ocean and sea-ice models.…”