A block-tridiagonal solver with two-level parallelization for finite element-spectral codes

Abstract

Abstract:Two--level parallelization is introduced to solve a massive block--tridiagonal matrix system. One--level is used for distributing blocks whose size is as large as the number of block rows due to the spectral basis, and the other level is used for parallelizing in the block row dimension. The purpose of the added parallelization dimension is to retard the saturation of the scaling due to communication overhead and inefficiencies in the single--level parallelization only distributing blocks. As a technique for parallelizing the tridiagonal matrix, the combined method of "Partitioned Thomas method" and "Cyclic Odd--Even Reduction" is implemented in an MPI--Fortran90 based finite element--spectral code (TORIC) that calculates the propagation of electromagnetic waves in a tokamak. The two--level parallel solver using thousands of processors shows more than 5 times improved computation speed with the optimized processor grid compared to the single--level parallel solver under the same conditions. Three--dimensional RF field reconstructions in a tokamak are shown as examples of the physics simulations that have been enabled by this algorithmic advance.

An Assessment Of Full Wave Effects On Maxwellian Lower Hybrid Wave Damping

“…A cubic Hermite interpolating polynomial finite element (FE) discretization is used in the radial direction ψ to solve for the Fourier electric field coefficients E (m) (ψ) on a flux surface. TORLH solves for a single toroidal mode n φ , but multiple simulations at different toroidal mode numbers may be superimposed to reconstruct a 3-D field [40]. The use of a finite element basis in the radial direction, however, makes TORLH significantly faster (N fewer operations) and less memory intensive (N 2 less memory) than full-spectral solvers such as AORSA [41], but the use of a Fourier basis along each flux surface in TORLH allows inclusion of hot plasma effects in a fully self-consistent manner as the Fourier basis properly accounts for non-locality in the dielectric response on the flux surface.…”

“…In TORLH, Helmholtz's equation ( 9) is put in Galerkin weak form and its dimension is reduced removing the radial field component E ψ (this reduction may be performed as we have ordered out the term ∝ σN 2 ⊥ corresponding to the pressure driven wave that occurs about the LH resonance and does not appear in LHCD scenarios [27]). Using discretization (5) a block tridiagonal matrix is produced which may be inverted to produce an electric field solution using a custom 3-D parallelized block-cyclic reduction solver [40].…”

“…is due to the good performance scaling of the matrix inversion algorithm and the memory constraints [40]. The matrix to be inverted in TORLH must be distributed and stored in the RAM throughout the inversion).…”

“…However, it was not possible to increase N m past 4095. For the TORLH solver to run effectively, N m must have a value of 2 n − 1 with integer n [40]. The computational resources required to perform N m = 8191 simulations do not exist as the memory requirement of TORLH solve scales with N 2 m and the simulations here, which used a large fraction of NERSC, were already memory limited.…”

Implicit numerical multidimensional heat-conduction algorithm parallelization and acceleration on a graphics card

“…It has been shown that a seemingly lesser algorithm can be much more efficient because it allows parallel processing [12][13][14][15] that is absolutely necessary for the use of the maximum computational power of modern processors, where thousands of tasks can be computed at the same time. The time when a personal computer's central processing unit (CPU) was able to compute only one thread at a time is over.…”