A cache-aware approach to domain decomposition for stencil-based codes

Abstract

Abstract-Partial Differential Equations (PDEs) lie at the heart of numerous scientific simulations depicting physical phenomena. The parallelization of such simulations introduces additional performance penalties in the form of local and global synchronization among cooperating processes. Domain decomposition partitions the largest shareable data structures into sub-domains and attempts to achieve perfect load balance and minimal communication. Up to now research efforts to optimize spatial and temporal cache reuse for stencil-based PDE discretizations (e.g. finite difference and finite element) have considered sub-domain operations after the domain decomposition has been determined. We derive a cache-oblivious heuristic that minimizes cache misses at the sub-domain level through a quasi-cache-directed analysis to predict families of high performance domain decompositions in structured 3-D grids. To the best of our knowledge this is the first work to optimize domain decompositions by analyzing cache misses -thus connecting single core parameters (i.e. cache-misses) to true multicore parameters (i.e. domain decomposition). We analyze the trade-offs in decreasing cache-misses through such decompositions and increasing the dynamic bandwidth-per-core. The limitation of our work is that currently, it is applicable only to structured 3-D grids with cuts parallel to the Cartesian Axes. We emphasize and conclude that there is an imperative need to re-think domain decompositions in this constantly evolving multicore era.

A quasi‐cache‐aware model for optimal domain partitioning in parallel geometric multigrid

“…We investigate the optimality of partitions returned by MPI_Dims_create() and whether only minimizing communication is sufficient to obtain optimal sub‐domain dimensions for parallel GMG. Our work in Saxena et al demonstrated the dependence of domain partitioning for single grids on cache‐misses in computation and communication. Since parallel GMG is significantly more complex than a single grid and incorporates further stencil operators, the current research examines the efficacy of extending the model to parallel GMG.…”

“…However, in some applications due to, e.g., varying coefficients, nonlinearities, or more complex grids, this may not be the case. Since data is mapped out linearly in memory regardless of the dimension of an array data structure, the non‐contiguous access pattern produced by stencils increases the cache‐misses . Efforts have been made to optimize and exploit spatial and temporal principles of the cache memory hierarchy to bridge the gap between the fast processor speed and the comparatively slower memory access times.…”

“…Each sub‐domain with a single element deep ghost/halo zone has dimension ( P x +2)( P y +2)( P z +2). The 3‐D sub‐domain on each core can be viewed as 3 parts: the inner Independent Computational (IC) kernel which needs no data from the neighbouring processes (zone 1), the next‐to‐boundary layer ( Dependent Planes ) which requires data from neighbouring processes for its update (zone 2), and the buffer/ghost/halo region (zone 3) . Thus, in the worst case, a sub‐domain will need to pass six planes to its nearest neighbour processes.…”

“…We now discuss the derivation of our quasi‐cache aware model that utilizes the cache‐line length and the data access pattern in a 7‐pt stencil. Our work in Saxena et al exhaustively identified and quantified cache‐misses as the single most important factor influencing domain partitioning of structured single level grids and thus, while extending the model, our focus remains on the cache‐misses in the update/packing/unpacking of the dependent planes and the update of the Independent Compute kernel. We further elaborate on the super‐set of factors influencing cache‐misses directly or indirectly to shed light on the complexity of attaining truly optimal sub‐domain dimensions for high performing partitions in Parallel Geometric Multigrid.…”

“…Since data is mapped out linearly in memory regardless of the dimension of an array data structure, the non-contiguous access pattern produced by stencils increases the cache-misses. 24 Efforts have been made to optimize and exploit spatial and temporal principles of the cache memory hierarchy to bridge the gap between the fast processor speed and the comparatively slower memory access times. [17][18][19][20][21]29 The Restriction and Interpolation inter-grid transfer operators in Multigrid are also Stencils.…”