Petascale direct numerical simulation of turbulent channel flow on up to 786K cores

Lee, Myoungkyu; Malaya, Nicholas; Moser, Robert D.

doi:10.1145/2503210.2503298

Cited by 80 publications

(47 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…(Lee et al, 2013) in BG/Q. Both codes are tested with grid size 3K×2K×1.5K in physical space using 1K MPI tasks and each task with 64 threads (16 cores with ×4 hyper-threading).…”

Section: A5 Performance Of the Code On Bg/q In Julich Supercomputimentioning

confidence: 99%

Coherent structures in statistically-stationary homogeneous shear turbulence

Dong¹

View full text Add to dashboard Cite

“…(Lee et al, 2013) in BG/Q. Both codes are tested with grid size 3K×2K×1.5K in physical space using 1K MPI tasks and each task with 64 threads (16 cores with ×4 hyper-threading).…”

Section: A5 Performance Of the Code On Bg/q In Julich Supercomputimentioning

confidence: 99%

Coherent structures in statistically-stationary homogeneous shear turbulence

Dong¹

View full text Add to dashboard Cite

“…Scaling will be efficient as long as the cost for MPI communication is smaller than all the other elementwise operations required to set up the right hand side of the explicit ODE given by Eq. (14). This is not a trivial task, considering that a pseudo-spectral solver demands that every process sends and receives data from all the other processes -at least for the slab decomposition.…”

Section: Parallel Scalingmentioning

confidence: 99%

High performance Python for direct numerical simulations of turbulent flows

Mortensen

Langtangen

2016

Computer Physics Communications

View full text Add to dashboard Cite

Direct Numerical Simulations (DNS) of the Navier Stokes equations is an invaluable research tool in fluid dynamics. Still, there are few publicly available research codes and, due to the heavy number crunching implied, available codes are usually written in low-level languages such as C/C++ or Fortran. In this paper we describe a pure scientific Python pseudo-spectral DNS code that nearly matches the performance of C++ for thousands of processors and billions of unknowns. We also describe a version optimized through Cython, that is found to match the speed of C++. The solvers are written from scratch in Python, both the mesh, the MPI domain decomposition, and the temporal integrators. The solvers have been verified and benchmarked on the Shaheen supercomputer at the KAUST supercomputing laboratory, and we are able to show very good scaling up to several thousand cores.A very important part of the implementation is the mesh decomposition (we implement both slab and pencil decompositions) and 3D parallel Fast Fourier Transforms (FFT). The mesh decomposition and FFT routines have been implemented in Python using serial FFT routines (either NumPy, pyFFTW or any other serial FFT module), NumPy array manipulations and with MPI communications handled by MPI for Python (mpi4py). We show how we are able to execute a 3D parallel FFT in Python for a slab mesh decomposition using 4 lines of compact Python code, for which the parallel performance on Shaheen is found to be slightly better than similar routines provided through the FFTW library. For a pencil mesh decomposition 7 lines of code is required to execute a transform.

show abstract

“…However, the Nelder-Mead method of AH is originally designed to work in a multi-dimensional orthotope (hyperrectangle) parameter space. 2 So the AH server can generate an infeasible test configuration. We customized the AH client to report the worst performance value (infinity) immediately back to the AH server when the AH client receives an infeasible configuration.…”

Section: Customizationmentioning

confidence: 99%

“…Bell et al [13] and Nishtala et al [8] wrote a UPC-based code that requires hardware support for asynchronous communication. As shown in Section 5, the UPC-based code is not optimized because (1) it misses some computation to overlap with communication, (2) it uses a fixed size of communication messages, and (3) it uses the point-to-point communication rather than the optimized collective operation. Fang et al [22] lack portability as they use a special communication API called QMP and require hardware support for asynchronous communication.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Computation–communication overlap and parameter auto-tuning for scalable parallel 3-D FFT

Song

Hollingsworth

2016

Journal of Computational Science

View full text Add to dashboard Cite

a b s t r a c tParallel 3-D FFT is widely used in scientific applications, therefore it is important to achieve high performance on large-scale systems with many thousands of computing cores. This paper describes a new method for scalable high-performance parallel 3-D FFT. We use a 2-D decomposition of 3-D arrays to increase scaling to a large number of cores. In order to achieve high performance, we use non-blocking MPI all-to-all operations and exploit computation-communication overlap. We also auto-tune our 3-D FFT code efficiently in a large parameter space and cope with the complex trade-off in optimizing our code in various system environments. According to experimental results from two systems, our method computes parallel 3-D FFT significantly faster than three existing libraries, and scales well to at least 32,768 compute cores.

show abstract

Petascale direct numerical simulation of turbulent channel flow on up to 786K cores

Cited by 80 publications

References 16 publications

Coherent structures in statistically-stationary homogeneous shear turbulence

Coherent structures in statistically-stationary homogeneous shear turbulence

High performance Python for direct numerical simulations of turbulent flows

Computation–communication overlap and parameter auto-tuning for scalable parallel 3-D FFT

Contact Info

Product

Resources

About