Near-global climate simulation at 1 km resolution: establishing a performance baseline on 4888 GPUs with COSMO 5.0

Abstract

Abstract. The best hope for reducing long-standing global climate model biases is by increasing resolution to the kilometer scale. Here we present results from an ultrahigh-resolution non-hydrostatic climate model for a near-global setup running on the full Piz Daint supercomputer on 4888 GPUs (graphics processing units). The dynamical core of the model has been completely rewritten using a domain-specific language (DSL) for performance portability across different hardware architectures. Physical parameterizations and diagnostics have been ported using compiler directives. To our knowledge this represents the first complete atmospheric model being run entirely on accelerators on this scale. At a grid spacing of 930 m (1.9 km), we achieve a simulation throughput of 0.043 (0.23) simulated years per day and an energy consumption of 596 MWh per simulated year. Furthermore, we propose a new memory usage efficiency (MUE) metric that considers how efficiently the memory bandwidth – the dominant bottleneck of climate codes – is being used.

Current and Emerging Time-Integration Strategies in Global Numerical Weather and Climate Prediction

“…Promising approaches for satisfying the latter condition are exponential time integrators [36,47]; (b) to overcome the overly restrictive time-step limitations of EBTI schemes combined with highly scalable horizontal discretizations, either through horizontal/vertical splitting (HEVI) [2,8,40] or through combining SISL PBTI methods with discontinuous Galerkin (DG) discretization [99]; and (c) to further the scalability and the adaptation of algorithms to emerging HPC architectures involving SE [32] or fully-implicit time-stepping approaches [113], and further through exploiting additional parallelism with time-parallel algorithms [33].…”

Can We Constrain Uncertainty in Hydrologic Cycle Projections?

“…At kilometer-scale grid spacings, atmospheric models start to explicitly simulate deep convection, which results in a significant improvement in capturing the frequency, intensity, and amount of precipitation (Prein et al, 2015). Initial work on global-scale convection-permitting modeling shows promising results that this goal is achievable (Fuhrer et al, 2018;Satoh et al, 2008;Schwitalla et al, 2018). While most of these simulations so-called convection-permitting simulations were performed over northern midlatitudes, initial results over the tropics also show very promising results in improving longstanding model biases associated with a too weak Hadley cell circulation (Hart et al, 2018) or misrepresented tropical wind patterns (Klocke et al, 2017).…”

“…The ultimate goal is to simultaneously simulate deep convection explicitly while also accounting for global energetic constraints; this will be possible with global-scale convection-permitting simulations. Initial work on global-scale convection-permitting modeling shows promising results that this goal is achievable (Fuhrer et al, 2018;Satoh et al, 2008;Schwitalla et al, 2018). These simulations will be much more computationally intensive than GCM simulations, and coupling them to other Earth system processes is farther from the horizon, but nonetheless, they can be leveraged to inform our use of GCMs.…”

A New Number Format for Ensemble Simulations

“…Some of the models—in particular those that use explicit time stepping schemes in horizontal direction, which only require local communication within a time step—are able to scale reasonably well up to the size of today's supercomputers. The community efforts could therefore be considered to be successful (Fuhrer et al, ; Müller et al, ). However, so far, efforts fail to leverage more than a couple of percent of peak floating point performance of available hardware.…”

“…On the other hand, a reduction of data that needs to be available in memory and cache or shared between processing units will almost certainly improve model performance and reduce energy consumption. This is acknowledged by the community, and new performance measures have been suggested that are not based on peak performance but rather on efficiency of memory use (Balaji et al, ; Fuhrer et al, ).…”