Acceleration of the IMplicit–EXplicit nonhydrostatic unified model of the atmosphere on manycore processors

Abstract

We present the acceleration of an IMplicit-EXplicit (IMEX) non-hydrostatic atmospheric model on manycore processors such as GPUs and Intel's MIC architecture. IMEX time integration methods sidestep the constraint imposed by the Courant-Friedrichs-Lewy condition on explicit methods through corrective implicit solves within each time step. In this work, we implement and evaluate the performance of IMEX on manycore processors relative to explicit methods. Using 3D-IMEX at Courant number C=15 , we obtained a speedup of about 4X relative to an explicit time stepping method run with the maximum allowable C=1. Moreover, the unconditional stability of IMEX with respect to the fast waves means the speedup can increase significantly with the Courant number as long as the accuracy of the resulting solution is acceptable. We show a speedup of 100X at C=150 using 1D-IMEX to demonstrate this point. Several improvements on the IMEX procedure were necessary in order to outperform our results with explicit methods: a) reducing the number of degrees of freedom of the IMEX formulation by forming the Schur complement; b) formulating a horizontally-explicit vertically-implicit (HEVI) 1D-IMEX scheme that has a lower workload and potentially better scalability than 3D-IMEX; c) using high-order polynomial preconditioners to reduce the condition number of the resulting system; d) using a direct solver for the 1D-IMEX method by performing and storing LU factorizations once to obtain a constant cost for any Courant number. Without all of these improvements, explicit time integration methods turned out to be difficult to beat. We discuss in detail the IMEX infrastructure required for formulating and implementing efficient methods on manycore processors. Several parametric studies are conducted to demonstrate the gain from each of the above mentioned improvements. Finally, we validate our results with standard benchmark problems in numerical weather prediction and evaluate the performance and scalability of the IMEX method using up to 4192 GPUs and 16 Knights Landing processors.

NCAR Release of CAM‐SE in CESM2.0: A Reformulation of the Spectral Element Dynamical Core in Dry‐Mass Vertical Coordinates With Comprehensive Treatment of Condensates and Energy

“…Over the last two decades or so, the spectral element (SE) method has been considered as a numerical method for the fluid flow solver in global weather/climate models (Baer et al, ; Choi & Hong, ; Fournier et al, ; Giraldo et al, ; Kelly & Giraldo, ). The main motivations were the SE methods' near‐perfect scalability (Dennis et al, ), GPU (Graphics Processing Unit) acceleration (e.g., Abdi, Giraldo, et al, ; Abdi, Wilcox, et al, ), high‐order accuracy for smooth problems, and mesh refinement capabilities. For some time the Community Earth System Model (CESM; Hurrell et al, ) has supported a SE dynamical core option in the atmosphere component CAM (Community Atmosphere Model; Neale et al, ) discretized on a cubed‐sphere grid (Figure a).…”

NPS-NRL-Rice-UIUC Collaboration on Navy Atmosphere-Ocean Coupled Models on Many-Core Computer Architectures Annual Report

“…The second assessment concerns the tests we conducted with OCCA kernels in the NUMA code on fully 1D and 3D Implicit-Explicit time-integrated dynamics. We describe in detail these results in the RESULTS section and present them in more detail in Abdi, Giraldo, Constantinescu, Carr III, Wilcox, and Warburton [4].…”

“…We have validated our results with standard benchmark problems in numerical weather prediction and evaluated the performance and strong scalability of the IMEX method using up to 4192 GPUs. The results regarding the implementation of the IMEX methods in NUMA on many-core architectures can be found in Abdi, Giraldo, Constantinescu, Carr III, Wilcox, and Warburton [4].…”

“…The largest time-to-solution obtained using 128 MPI processes is about 7% slower than one using purely OpenMP with 128 threads. More details on running NUMA using implicit time-integration on either Nvidia GPUs or Intel's KNL can be found in Abdi, Giraldo, Constantinescu, Carr III, Wilcox, and Warburton [4].…”

Current and Emerging Time-Integration Strategies in Global Numerical Weather and Climate Prediction

“…Promising approaches for satisfying the latter condition are exponential time integrators [36,47]; (b) to overcome the overly restrictive time-step limitations of EBTI schemes combined with highly scalable horizontal discretizations, either through horizontal/vertical splitting (HEVI) [2,8,40] or through combining SISL PBTI methods with discontinuous Galerkin (DG) discretization [99]; and (c) to further the scalability and the adaptation of algorithms to emerging HPC architectures involving SE [32] or fully-implicit time-stepping approaches [113], and further through exploiting additional parallelism with time-parallel algorithms [33].…”