H-AMR: A New GPU-accelerated GRMHD Code for Exascale Computing With 3D Adaptive Mesh Refinement and Local Adaptive Time-stepping

Abstract

General-relativistic magnetohydrodynamic (GRMHD) simulations have revolutionized our understanding of black-hole accretion. Here, we present a GPU-accelerated GRMHD code H-AMR with multi-faceted optimizations that, collectively, accelerate computation by 2-5 orders of magnitude for a wide range of applications. Firstly, it involves a novel implementation of a spherical-polar grid with 3D adaptive mesh refinement that operates in each of the 3 dimensions independently. This allows us to circumvent the Courant condition near the polar singularity, which otherwise cripples high-res computational performance. Secondly, we demonstrate that local adaptive time-stepping (LAT) on a logarithmic spherical-polar grid accelerates computation by a factor of 10 compared to traditional hierarchical time-stepping approaches. Jointly, these unique features lead to an effective speed of ∼ 10 9 zone-cyclesper-second-per-node on 5,400 NVIDIA V100 GPUs (i.e., 900 nodes of the OLCF Summit supercomputer). We demonstrate its computational performance by presenting the first GRMHD simulation of a tilted thin accretion disk threaded by a toroidal magnetic field around a rapidly spinning black hole. With an effective resolution of 13,440 × 4,608 × 8,092 cells, and a total of 22 billion cells and ∼ 0.65 × 10 8 timesteps, it is among the largest astrophysical simulations ever performed. We find that frame-dragging by the black hole tears up the disk into two independently precessing sub-disks. The innermost sub-disk rotation axis intermittently aligns with the black hole spin, demonstrating for the first time that such long-sought alignment is possible in the absence of large-scale poloidal magnetic fields.

Black hole flares: ejection of accreted magnetic flux through 3D plasmoid-mediated reconnection

“…In the plasmoid-mediated regime, the reconnection rate converges to the asymptotic v rec ∼ 0.01c in GRMHD (Ripperda et al 2020;Bransgrove et al 2021), directly determining the (converged) rate of magnetic flux decay on the horizon. To achieve the resolution required to capture the plasmoid-mediated reconnection and, hence, achieve long-sought convergence in the reconnection rate, we employ our GPU-accelerated GRMHD code H-AMR (Liska et al 2019). We set the effective numerical resolution to N r ×N θ ×N φ = 5376×2304×2304 (dubbed "extreme resolution" from here onward) to ensure that we capture thin plasmoid-unstable current sheets (Ripperda et al 2020).…”

Jet Formation in 3D GRMHD Simulations of Bondi-Hoyle-Lyttleton Accretion

“…We evolve our simulations using the GPU-accelerated, three-dimensional general-relativistic magnetohydrodynamic code H-AMR (Liska et al 2019). H-AMR uses a spherical polar grid that is uniform in logr and centered on the BH.…”

“…We also use multiple numerical speed-ups that make H-AMR well-suited to problems with large scale separation, such as the one studied here. This includes five levels of local adaptive time-stepping and θ-dependent static refinement criteria (for details see Liska et al 2019). The base resolution of our grid is (N r × N θ × N φ ) = (768 × 192 × 256).…”

Black hole to breakout: 3D GRMHD simulations of collapsar jets reveal a wide range of transients

“…To test and calibrate the analytic assumptions in §2 we run a suite of GRMHD simulations of self-consistent jet launching in collapsars, using the 3D GPU-accelerated code H-AMR (Liska et al 2019). For the simulated collapsars we take a WR-like star with a radius R = 4 × 10 10 cm and mass M ≈ 14 M .…”