Dynamic Data Migration for Structured AMR Solvers

Abstract

On cc-NUMA multi-processors, the non-uniformity of main memory latencies motivates the need for co-location of threads and data. We call this special form of data locality, geographical locality. In this article, we study the performance of a parallel PDE solver with adaptive mesh refinement (AMR). The solver is parallelized using OpenMP and the adaptive mesh refinement makes dynamic load balancing necessary. Due to the dynamically changing memory access pattern caused by the runtime adaption, it is a challenging task to achieve a high degree of geographical locality. The main conclusions of the study are: (1) that geographical locality is very important for the performance of the solver, (2) that the performance can be improved significantly using dynamic page migration of misplaced data, (3) that a migrate-onnext-touch directive works well whereas the first-touch strategy is less advantageous for programs exhibiting a dynamically changing memory access patterns, and (4) that the overhead for such migration is low compared to the total execution time.

Mapping applications for high performance on multithreaded, NUMA systems

“…Replication and distribution are two main techniques to bring data close to CPUs on NUMA machines (e.g., [2,8,7]). …”

“…Geographical locality (e.g., see [4,8]) was proposed on NUMA systems in addition to spatial and temporal locality as a measure of the distance between data and computation. Prior studies (e.g., see [2,8]) optimize geographical locality through data replication and distribution. These techniques have serious limitations for irregular applications.…”

Lessons Learnt Porting Parallelisation Techniques for Irregular Codes to NUMA Systems

“…Authors examine a wide variety of techniques including the influence of the process and memory affinity. Regarding memory allocation, Norden et al [6] study the co-location of threads and data motivated by the nonuniformity of memory in NUMA multi-processors.…”

Effective Use of Dynamic Page Migration on NUMA Platforms: The Gaussian Chemistry Code on the SunFire X4600M2 System

“…This leads to improved data locality, although the migration process is not without overhead. Dynamic page migration has been used in a range of applications [4], [5], [6].…”