Optimizing the HOMME dynamical core for multicore platforms

Abstract

The approach of the next-generation computing platforms offers a tremendous opportunity to advance the state-of-the-art in global atmospheric dynamical models. We detail our incremental approach to utilize this emerging technology by enhancing concurrency within the High-Order Method Modeling Environment (HOMME) atmospheric dynamical model developed at the National Center for Atmospheric Research (NCAR). The study focused on improvements to the performance of HOMME which is a Fortran 90 code with a hybrid (MPIOpenMP) programming model. The article describes the changes made to the use of message passing interface (MPI) and OpenMP as well as single-core optimizations to achieve significant improvements in concurrency and overall code performance. For our optimization studies, we utilize the “Cori” system with an Intel Xeon Phi Knights Landing processor deployed at the National Energy Research Supercomputing Center and the “`Cheyenne” system with an Intel Xeon Broadwell processor installed at the NCAR. The results from the studies, using “workhorse” configurations performed at NCAR, show that these changes have a transformative impact on the computational performance of HOMME. Our improvements have shown that we can effectively increase potential concurrency by efficiently threading the vertical dimension. Further, we have seen a factor of two overall improvement in the computational performance of the code resulting from the single-core optimizations. Most notably from the work is that our incremental approach allows for high-impact changes without disrupting existing scientific productivity in the HOMME community.

Technical descriptions of the experimental dynamical downscaling simulations over North America by the CAM5.4-MPAS4.0 variable-resolution model

“…Lastly, we get consistently lower throughput and higher costs on Cori-KNL than on the other two systems. Our experiment and previous studies (Barnes et al, 2017;Dennis et al, 2019) suggest a few compounding reasons (Appendix D): inefficient memory management for some global arrays, poor vectorization, and less focus on shared-memory parallelism of the CAM5/MPASv4 source code, which are not aligned well with the wider-vector and many-core architecture of KNL. However, the shorter expected queue time on KNL than HW (Figure D1) makes KNL our main system for production.…”

“…Participation of VR models allows direct and more comprehensive intercomparison of limited-area and global VR models, but requires appropriate adaptations of the experimental protocol and analysis scope to address differences between the two modeling framework, such as the evaluation of soil state and large-scale circulations outside the refinement domain. Having both limited-area and VR models in a coordinated project The overall performance of climate model code is typically limited by memory latency and bandwidth rather than arithmetic speed (e.g., Fuhrer et al, 2018;Dennis et al, 2019), except for some components such as the MG2 microphysics (Barnes et al, 2017). A naive use of all the 68 cores on KNL nodes as MPI ranks lead to 0.5 GB memory per rank (using KNL's two different memory units as a single entity), about one-fifth of 2.7 GB per rank when using 24 MPI ranks per node on Edison.…”

“…The MG2 microphysics code was found to be bounded by computation with poor vectorization, and improved code structure for easier vectorization enhanced its speed by about 75% (He, 2016;Barnes et al, 2017). Optimizations of the High Order Methods Modeling Environment (HOMME) dynamical core involved both better vectorization and rewriting the OpenMP loops, which together achieved twice faster performance on KNL (Barnes et al, 2017;Dennis et al, 2019). While some optimizations are more specific to KNL, many of the code changes improve performance on other systems such as the Cheyenne system in the National Center for Atmospheric Research -Wyoming Supercomputing Center (Dennis et al, 2019).…”