On the communication complexity of 3D FFTs and its implications for Exascale

Czechowski, Kenneth; Battaglino, Casey; McClanahan, Chris; Iyer, Kartik P.; Yeung, P. K.; Vuduc, Richard

doi:10.1145/2304576.2304604

Cited by 50 publications

(44 citation statements)

References 36 publications

(30 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Pippig [14] reported a comparative study of FFTW [15], PFFT [14], and P3DFFT [16] using an IBM Blue Gene machine. Similar investigations using Intel-Infiniband clusters were reported in [13,17,18]. In general, the majority of the execution time is spent in communication.…”

Section: Introductionsupporting

confidence: 80%

Spectral Domain Decomposition Using Local Fourier Basis: Application to Ultrasound Simulation on a Cluster of GPUs

Jaroš

Vaverka

Treeby

2016

JSFI

View full text Add to dashboard Cite

The simulation of ultrasound wave propagation through biological tissue has a wide range of practical applications. However, large grid sizes are generally needed to capture the phenomena of interest. Here, a novel approach to reduce the computational complexity is presented. The model uses an accelerated k-space pseudospectral method which enables more than one hundred GPUs to be exploited to solve problems with more than 3 ×10 9 grid points. The classic communication bottleneck of Fourier spectral methods, all-to-all global data exchange, is overcome by the application of domain decomposition using local Fourier basis. Compared to global domain decomposition, for a grid size of 1536 × 1024 × 2048, this reduces the simulation time by a factor of 7.5 and the simulation cost by a factor of 3.8.

show abstract

Section: Introductionsupporting

confidence: 80%

Spectral Domain Decomposition Using Local Fourier Basis: Application to Ultrasound Simulation on a Cluster of GPUs

Jaroš

Vaverka

Treeby

2016

JSFI

View full text Add to dashboard Cite

show abstract

“…) cache misses, for each transferring line of size L [13,14]. This bound is optimal, matching the lower bound by Hong and Kung [15] when 3 √ N is an exact power of two.…”

Section: Memory Access Costssupporting

confidence: 71%

FFT, FMM, and multigrid on the road to exascale: Performance challenges and opportunities

Ibeid

Olson

Gropp

2020

Journal of Parallel and Distributed Computing

View full text Add to dashboard Cite

FFT, FMM, and multigrid methods are widely used fast and highly scalable solvers for elliptic PDEs. However, emerging large-scale computing systems are introducing challenges in comparison to current petascale computers. Recent efforts [1] have identified several constraints in the design of exascale software that include massive concurrency, resilience management, exploiting the high performance of heterogeneous systems, energy efficiency, and utilizing the deeper and more complex memory hierarchy expected at exascale. In this paper, we perform a model-based comparison of the FFT, FMM, and multigrid methods in the context of these projected constraints. In addition we use performance models to offer predictions about the expected performance on upcoming exascale system configurations based on current technology trends.

show abstract

“…We use the AccFFT package [45]-a parallel, open-source FFT library for CPU/GPU architectures developed in our group, to apply the spectral operators. AccFFT dictates the data layout: We partition the data based on a pencil decomposition for 3D FFTs [35,50]: Let n p = p 1 p 2 denote the number of MPI tasks; then each MPI task gets (n 1 /p 1 ) × (n 2 /p 2 ) × n 3 grid points (i.e., we partition the domain Ω h along the x 1 -and x 2 -axis into subdomains Ω h i , i = 1, . .…”

Section: Newtonmentioning

confidence: 99%

CLAIRE: A Distributed-Memory Solver for Constrained Large Deformation Diffeomorphic Image Registration

Mang

Gholami

Davatzikos

et al. 2019

SIAM J. Sci. Comput.

112

View full text Add to dashboard Cite

We introduce CLAIRE, a distributed-memory algorithm and software for solving constrained large deformation diffeomorphic image registration problems in three dimensions. We invert for a stationary velocity field that parameterizes the deformation map. Our solver is based on a globalized, preconditioned, inexact reduced space Gauss-Newton-Krylov scheme.We exploit state-of-the-art techniques in scientific computing to develop an effective solver that scales to thousand of distributed memory nodes on high-end clusters. Our improved, parallel implementation features parameter-, scale-, and grid-continuation schemes to speedup the computations and reduce the likelihood to get trapped in local minima. We also implement an improved preconditioner for the reduced space Hessian to speedup the convergence.We test registration performance on synthetic and real data. We demonstrate registration accuracy on 16 neuroimaging datasets. We compare the performance of our scheme against different flavors of the DEMONS algorithm for diffeomorphic image registration. We study convergence of our preconditioner and our overall algorithm. We report scalability results on state-of-the-art supercomputing platforms. We demonstrate that we can solve registration problems for clinically relevant data sizes in two to four minutes on a standard compute node with 20 cores, attaining excellent data fidelity. With the present work we achieve a speedup of (on average) 5× with a peak performance of up to 17× compared to our former work. Introduction.Deformable registration is a key technology in the medical imaging. It is about computing a map y that establishes a meaningful spatial correspondence between two (or more) images m R (the reference (fixed) image) and m T (the template (deformable or moving) image; image to be registered) of the same scene [43,96]. Numerous approaches for formulating and solving image registration problems have appeared in the past; we refer to [43,60,96,97,116] for lucid overviews. Image registration is typically formulated as a variational optimization problem that consists of a data fidelity term and a Tikhonov regularization functional to over-come ill-posedness [40,43]. A key concern is that y is a diffeomorphism, i.e., the map y is differentiable, a bijection, and has a differentiable inverse. We require that the determinant of the deformation gradient det ∇y does not vanish or change sign. An intuitive approach to safeguard against nondiffeomorphic maps y is to add hard and/or soft constraints on det ∇y to the variational optimization problem [29,58,103,108]. An alternative strategy to ensure regularity is to introduce a pseudo-time variable t and invert for a smooth velocity field v that encodes the map y [16,37,94,126]; existence of a diffeomorphism y can be guaranteed if v is adequately smooth [16,30,37,121]. This model is, as opposed to many traditional formulations that directly invert for y [58,88,95,108], adequate for recovering large, highly nonlinear deformations. Our approach falls into this category. T...

show abstract

On the communication complexity of 3D FFTs and its implications for Exascale

Cited by 50 publications

References 36 publications

Spectral Domain Decomposition Using Local Fourier Basis: Application to Ultrasound Simulation on a Cluster of GPUs

Spectral Domain Decomposition Using Local Fourier Basis: Application to Ultrasound Simulation on a Cluster of GPUs

FFT, FMM, and multigrid on the road to exascale: Performance challenges and opportunities

CLAIRE: A Distributed-Memory Solver for Constrained Large Deformation Diffeomorphic Image Registration

Contact Info

Product

Resources

About