Large-scale 3-D EM modelling with a Block Low-Rank multifrontal direct solver

Abstract

S U M M A R YWe put forward the idea of using a Block Low-Rank (BLR) multifrontal direct solver to efficiently solve the linear systems of equations arising from a finite-difference discretization of the frequency-domain Maxwell equations for 3-D electromagnetic (EM) problems. The solver uses a low-rank representation for the off-diagonal blocks of the intermediate dense matrices arising in the multifrontal method to reduce the computational load. A numerical threshold, the so-called BLR threshold, controlling the accuracy of low-rank representations was optimized by balancing errors in the computed EM fields against savings in floating point operations (flops). Simulations were carried out over large-scale 3-D resistivity models representing typical scenarios for marine controlled-source EM surveys, and in particular the SEG SEAM model which contains an irregular salt body. The flop count, size of factor matrices and elapsed run time for matrix factorization are reduced dramatically by using BLR representations and can go down to, respectively, 10, 30 and 40 per cent of their full-rank values for our largest system with N = 20.6 million unknowns. The reductions are almost independent of the number of MPI tasks and threads at least up to 90 × 10 = 900 cores. The BLR savings increase for larger systems, which reduces the factorization flop complexity from O(N 2 ) for the full-rank solver to O(N m ) with m = 1.4-1.6. The BLR savings are significantly larger for deep-water environments that exclude the highly resistive air layer from the computational domain. A study in a scenario where simulations are required at multiple source locations shows that the BLR solver can become competitive in comparison to iterative solvers as an engine for 3-D controlled-source electromagnetic Gauss-Newton inversion that requires forward modelling for a few thousand right-hand sides.

Efficient use of sparsity by direct solvers applied to 3D controlled-source EM problems

“…We now summarize the new results obtained on the set of systems presented in Table 1, and show that the work described in this article has a large impact on the global resolution times. We also relate the results to previous work (Shantsev et al, 2017) which compared the direct approach to an iterative one.…”

“…We also report in Table 1 the analysis, factorization and solve times of the sparse direct solver MUMPS using BLR compression (Amestoy et al, 2018) on the CALMIP supercomputer EOS (https://www.calmip.univ-toulouse.fr/), which is a BULLx DLC system composed of 612 computing nodes, each composed of two Intel Ivybridge processors with 10 cores (total 12 240 cores) running at 2.8 GHz, with 64 GBytes of memory per node. As mentioned earlier, the introduction of low-rank approximations has significantly reduced the factorization time (Shantsev et al, 2017), and the initial solve time T s (not using the work presented in this paper) has become predominant. Note that the solve phase was performed by blocks of size BLK = 1024 for H3 and BLK = 512 for H17, S21 and DB30.…”

“…Properties of system matrices and right-hand sides resulting from these discretizations are summarized in Table 1. The matrices H3, H17 and S21 are described in detail in (Shantsev et al, 2017). The two Hmatrices are based on a half-space 1 Ωm model with a 100 m water layer and a pizza-box resistor of 100 Ωm.…”

“…The complexity of the factorization phase can be a bottleneck for large 3D problems since the number of floating-point operations scales as O(n 2 ) and the number of nonzero entries in the factors is O(n 4/3 ) which also defines the complexity of the solve phase. However, in the context of CSEM applications, it has been shown (Shantsev et al, 2017) that using Block Low-Rank (BLR) format and related approximations significantly improves the performance of the direct approach and reduces the factorization complexity from O(n 2 ) to O(n 4/3 ) for a simple block low-rank (BLR) format . Thanks to the improvements of the factorization phase due to low rank compression (further improved in (Amestoy et al, 2018) with respect to (Shantsev et al, 2017)), and since CSEM modeling involves thousands of right-hand sides, the time needed to perform the complete CSEM simulation becomes largely dominated by the solve phase of the direct solver.…”

Deep learning electromagnetic inversion with convolutional neural networks

“…Modern seismic and EM surveys include hundreds and thousands of sources and receivers, thus significantly increasing the computational cost of forward modelling. Accurate simulations of EM phenomena require fine-scale models and lead to forward problems with tens of millions of unknowns (Shantsev et al, 2017). Consequently, inversion of a large geophysical dataset using traditional deterministic approaches requires the solution of several millions of forward problems each having up to tens of millions of unknowns.…”