A scalability study of the Ice-sheet and Sea-level System Model (ISSM, version 4.18)

Abstract

Abstract. Accurately modelling the contribution of Greenland and Antarctica to sea level rise requires solving partial differential equations at a high spatial resolution. In this paper, we discuss the scaling of the Ice-sheet and Sea-level System Model (ISSM) applied to the Greenland Ice Sheet with horizontal grid resolutions varying between 10 and 0.25 km. The model setup used as benchmark problem comprises a variety of modules with different levels of complexity and computational demands. The core builds the so-called stress balance module, which uses the higher-order approximation (or Blatter–Pattyn) of the Stokes equations, including free surface and ice-front evolution as well as thermodynamics in form of an enthalpy balance, and a mesh of linear prismatic finite elements, to compute the ice flow. We develop a detailed user-oriented, yet low-overhead, performance instrumentation tailored to the requirements of Earth system models and run scaling tests up to 6144 Message Passing Interface (MPI) processes. The results show that the computation of the Greenland model scales overall well up to 3072 MPI processes but is eventually slowed down by matrix assembly, the output handling and lower-dimensional problems that employ lower numbers of unknowns per MPI process. We also discuss improvements of the scaling and identify further improvements needed for climate research. The instrumented version of ISSM thus not only identifies potential performance bottlenecks that were not present at lower core counts but also provides the capability to continually monitor the performance of ISSM code basis. This is of long-term significance as the overall performance of ISSM model depends on the subtle interplay between algorithms, their implementation, underlying libraries, compilers, runtime systems and hardware characteristics, all of which are in a constant state of flux. We believe that future large-scale high-performance computing (HPC) systems will continue to employ the MPI-based programming paradigm on the road to exascale. Our scaling study pertains to a particular modelling setup available within ISSM and does not address accelerator techniques such as the use of vector units or GPUs. However, with 6144 MPI processes, we identified issues that need to be addressed in order to improve the ability of the ISSM code base to take advantage of upcoming systems that will require scaling to even higher numbers of MPI processes.

Performance analysis of high-resolution ice-sheet simulations

“…Indeed, many models (e.g. Leng and others, 2012;Brown and others, 2013;Isaac and others, 2015;Tuminaro and others, 2016;Fischler and others, 2022) also show good parallel scaling, something not addressed here.…”

“…Some hybrid schemes apply the pessimistic paradigm as an adaptive restriction (Winkelmann and others, 2011) across all spatial scales. Other models apparently require the user to choose a fixed time step through trial and error in some circumstances (Fischler and others, 2022; Robinson and others, 2022, e.g.). The optimistic paradigm has theoretical support for a certain higher-order DIVA scheme (Robinson and others, 2022, Equation (52)), but practical Greenland simulations in the same work actually suggest an intermediate power Δ t ≈ O (Δ x 1.6 ) (Robinson and others, 2022, Figure 3(a)).…”

“…Resolving the physics of marine ice sheets also benefits from fine resolution, although careful choice of basal parameterizations will reduce resolution dependence (Gladstone and others, 2017). Thus, whether using local mesh refinement (Hoffman and others, 2018; Fischler and others, 2022, e.g.) or not, resolutions of 2 or 1 km (Seguinot and Delaney, 2021) or less (Clarke and others, 2015; Aschwanden and others, 2019) are increasingly used for science at ice-sheet scale.…”

A parallel implementation of the confined-unconfined aquifer system model for subglacial hydrology: design, verification, and performance analysis (CUAS-MPI v0.1.0)

“…For this endeavour, it is worth comparing the computational costs of the ice sheet and hydrology models. Comparing the SYPD for a Greenland setup in the ice sheet model in the highest tested resolution (G250) presented in (Fischler et al, 2022) with the G150 resolution of CUAS-MPI, we find that the computional costs are comparable for both models. As a consequence, both simulations would need the same time per year, which which is preferable for achieving low idle time at synchronization points in coupled runs.…”

Performance portable ice-sheet modeling with MALI

“…The computational component of the hybrid "SSA+SIA" model in PISM is found to scale well on up to 1024 cores in [17] but the I/O component is found to scale poorly. In [21], low-overhead performance instrumentation is developed for the "Blatter-Pattyn" model in ISSM and good scaling is found on up to 3072 cores. The study finds that matrix assembly and I/O begins to scale poorly and highlights the importance of continuous performance monitoring.…”