SpECTRE: A task-based discontinuous Galerkin code for relativistic astrophysics

Abstract

We introduce a new relativistic astrophysics code, SpECTRE, that combines a discontinuous Galerkin method with a task-based parallelism model. SpECTRE's goal is to achieve more accurate solutions for challenging relativistic astrophysics problems such as core-collapse supernovae and binary neutron star mergers. The robustness of the discontinuous Galerkin method allows for the use of high-resolution shock capturing methods in regions where (relativistic) shocks are found, while exploiting high-order accuracy in smooth regions. A taskbased parallelism model allows efficient use of the largest supercomputers for problems with a heterogeneous workload over disparate spatial and temporal scales. We argue that the locality and algorithmic structure of discontinuous Galerkin methods will exhibit good scalability within a task-based parallelism framework. We demonstrate the code on a wide variety of challenging benchmark problems in (non)-relativistic (magneto)hydrodynamics. We demonstrate the code's scalability including its strong scaling on the NCSA Blue Waters supercomputer up to the machine's full capacity of 22, 380 nodes using 671, 400 threads. variety of astrophysics codes (e.g., Refs. [6,[9][10][11][12][13][14]) have been designed based on these fundamental building blocks.These strategies work well when the computations are reasonably homogeneous or when one seeks good parallelization to only a few thousand cores. As the number of MPI processes increases, so does the cost of communication which, together with non-uniform workload typical of astrophysics problems, limits the maximum number of useful cores that codes can run on. Efficient core utilization becomes non-trivial, often requiring careful optimization by hand to achieve good scalability [15]. Standard finite-volume and finitedifference methods achieve higher order accuracy with increasingly large (overlapping) stencil sizes, and may require additional effort to achieve scalability on massively parallel machines.As one looks ahead to the arrival of exascale computing, it will become increasingly important to focus on developing algorithms that can take full advantage of these very large machines.Discontinuous Galerkin (DG) methods [16-21], together with a task-based parallelization strategy, have the potential to tackle many of these problems. DG methods offer high-order accuracy in smooth regions (although, for stability, increasing the scheme's order requires decreasing the timestep, which restricts the largest usable order in practice), robustness for shocks and other discontinuities, and grid flexibility including a formulation that allows for comparatively straightforward hp-adaptivity and local timestepping. DG methods can be combined with positivity preserving strategies [22][23][24] or "atmosphere treatments" [25] which seek to maintain non-negative values of the pressure and density in challenging regions such as those containing high-speed astrophysical flow. DG methods are also well suited for parallelization: Their formulation in terms of l...

High-order magnetohydrodynamics for astrophysics with an adaptive mesh refinement discontinuous Galerkin scheme

“…Gaburov & Nitadori 2011;Mignone et al 2012;Tricco & Price 2012;Hopkins & Raives 2016) including in a DG context (e.g. Boscheri et al 2014;Zanotti et al 2015;Dumbser & Loubère 2016;Kidder et al 2017). Our implementation closely follows the so-called generalized Lagrange multiplier (GLM) formulation of Dedner et al (2002).…”

“…These properties make DG schemes attractive for fixed grid Eulerian astrophysical codes, where large structures such as galaxies may travel across the grid with large bulk advection velocities. As a result, there has been significant recent interest in DG methods in astrophysics, for hydrodynamics (Schaal et al 2015;Velasco Romero et al 2018), ideal magnetohydrodynamics (Mocz et al 2014;Dumbser & Loubère 2016;Karami Halashi & Luo 2016), non-ideal MHD (Boscheri & Dumbser 2017), as well as special-and general-relativistic MHD (Zanotti et al 2015;Kidder et al 2017;Anninos et al 2017;Zhao & Tang 2017;Fambri et al 2018).…”

hp-adaptive discontinuous Galerkin solver for elliptic equations in numerical relativity

“…This can be amended by incorporating a task-based parallelism framework for load balancing. The elliptic solver developed in this article will be incorporated task-based parallel code SpECTRE, which is concurrently being developed [21].…”

“…The present work also complements the DG evolution code presented in Ref. [21], leading to a complete framework for solving both elliptic and hyperbolic PDEs in numerical relativity.…”

“…Discontinuous Galerkin (DG) methods [1][2][3][4][5][6] have matured into standard numerical methods to simulate a wide variety of partial differential equations. In the context of numerical relativity [7], discontinuous Galerkin methods have shown advantages for relativistic hyperbolic problems over traditional discretization methods such as finite difference, finite volume and spectral finite elements [8][9][10][11][12][13][14][15][16][17][18][19][20][21][22] by combining the best aspects of all three methods. As computers reach exa-scale power, new methods like DG are needed to tackle the biggest problems in numerical relativity such as realistic supernovae and binary neutron-star merger simulations on these very large machines [21].…”