Massive-Parallel Trajectory Calculations version 2.2 (MPTRAC-2.2): Lagrangian transport simulations on graphics processing units (GPUs)

Abstract

Abstract. Lagrangian models are fundamental tools to study atmospheric transport processes and for practical applications such as dispersion modeling for anthropogenic and natural emission sources. However, conducting large-scale Lagrangian transport simulations with millions of air parcels or more can become rather numerically costly. In this study, we assessed the potential of exploiting graphics processing units (GPUs) to accelerate Lagrangian transport simulations. We ported the Massive-Parallel Trajectory Calculations (MPTRAC) model to GPUs using the open accelerator (OpenACC) programming model. The trajectory calculations conducted within the MPTRAC model were fully ported to GPUs, i.e., except for feeding in the meteorological input data and for extracting the particle output data, the code operates entirely on the GPU devices without frequent data transfers between CPU and GPU memory. Model verification, performance analyses, and scaling tests of the Message Passing Interface (MPI) – Open Multi-Processing (OpenMP) – OpenACC hybrid parallelization of MPTRAC were conducted on the Jülich Wizard for European Leadership Science (JUWELS) Booster supercomputer operated by the Jülich Supercomputing Centre, Germany. The JUWELS Booster comprises 3744 NVIDIA A100 Tensor Core GPUs, providing a peak performance of 71.0 PFlop s−1. As of June 2021, it is the most powerful supercomputer in Europe and listed among the most energy-efficient systems internationally. For large-scale simulations comprising 108 particles driven by the European Centre for Medium-Range Weather Forecasts' fifth-generation reanalysis (ERA5), the performance evaluation showed a maximum speed-up of a factor of 16 due to the utilization of GPUs compared to CPU-only runs on the JUWELS Booster. In the large-scale GPU run, about 67 % of the runtime is spent on the physics calculations, conducted on the GPUs. Another 15 % of the runtime is required for file I/O, mostly to read the large ERA5 data set from disk. Meteorological data preprocessing on the CPUs also requires about 15 % of the runtime. Although this study identified potential for further improvements of the GPU code, we consider the MPTRAC model ready for production runs on the JUWELS Booster in its present form. The GPU code provides a much faster time to solution than the CPU code, which is particularly relevant for near-real-time applications of a Lagrangian transport model.

Lagrangian transport simulations using the extreme convection parametrization: an assessment for the ECMWF reanalyses

“…Code and data availability. The MPTRAC model (Hoffmann et al, 2016(Hoffmann et al, , 2022a is made available under the terms and conditions of the GNU General Public License (GPL) version 3. The release version 2.4 of MPTRAC applied in this paper has been archived on Zenodo (Hoffmann et al, 2022b).…”

“…MPTRAC provides various output methods for the particle data, including grid-averaged output, which is applied in this study. While MPTRAC can be applied on a single workstation, it features an MPI-OpenMP-OpenACC hybrid parallelization for efficient use on high performance computing (HPC) systems and graphics processing units (GPUs) (Rößler et al, 2018;Liu et al, 2020;Hoffmann et al, 2022a).…”

“…1 shows monthly mean CAPE, CIN, LFC, and EL data in July 2017 from the ERA5 reanalysis. The data have been calculated directly from the ERA5 temperature and specific humidity vertical profiles using the meteorological data preprocessing code of MPTRAC (Hoffmann et al, 2022a, value of an air parcel is larger than a threshold CAPE 0 , i. e., a user-defined control parameter of the ECP, and the air parcel is located below the EL, it is assumed that up-and downdrafts within the vertical column are strong enough to trigger a convective event. In the second step, the air parcels involved in a convective event are randomly redistributed in the vertical 160 column stretching from the surface to the EL.…”

The influence of the Asian summer monsoon on volcanic aerosol transport in the UTLS region

“…The transport and dispersion of volcanic plumes are simulated with the Massive-Parallel Trajectory Calculations model (MPTRAC) Version 2.2. MPTRAC is a Lagrangian particle dispersion model (LPDM) suitable for analyzing transport processes in the troposphere and stratosphere 47,48 . It uses wind fields from reanalyzed data to calculate the trajectories of particles by solving the kinematic equation of motion.…”

Improving trajectory calculations by FLEXPART 10.4+ using deep learning inspired single image superresolution

“…However, error reductions for higher-order schemes are less than 30%, while computational costs increase by about an order of magnitude [32]. Therefore, many trajectory and Lagrangian particle dispersion models still use linear interpolation, e.g., FLEXPART [21], LAGRANTO [30], or MPTRAC [11] The purpose of this paper is to implement variable-scale superresolution based on deep convolutional neural networks to showcase their potential for Lagrangian models. Here, we make use of the selfsimilarity of meteorological fields, such that a neural network can be repeatedly applied to interpolate a velocity field to higher resolutions.…”