A Hierarchical Data-Partitioning Algorithm for Performance Optimization of Data-Parallel Applications on Heterogeneous Multi-Accelerator NUMA Nodes

Abstract

Modern HPC platforms are highly heterogeneous with tight integration of multicore CPUs and accelerators (such as Graphics Processing Units, Intel Xeon Phis, or Field-Programmable Gate Arrays) empowering them to address the twin critical concerns of performance and energy efficiency. Due to this inherent characteristic, processing elements contend for shared on-chip resources such as Last Level Cache (LLC), interconnect, etc. and shared nodal resources such as DRAM, PCI-E links, etc., resulting in complexities such as resource contention, non-uniform memory access (NUMA), and accelerator-specific limitations such as limited main memory thereby necessitating support for efficient out-of-card execution. Due to these complexities, the performance profiles of data-parallel applications executing on these platforms are not smooth and deviate significantly from the shapes that allowed state-of-the-art load-balancing algorithms to find optimal solutions. In this paper, we propose a hierarchical two-level data partitioning algorithm minimizing the parallel execution time of data-parallel applications on clusters of h identical nodes where each node has c heterogeneous processors. This algorithm takes as input c discrete speed functions of cardinality m corresponding to the c heterogeneous processors. It does not make any assumptions about the shapes of these functions. Unlike load balancing algorithms, optimal solutions found by the algorithm may not load-balance an application in terms of execution time. The proposed algorithm has low time complexity of O(m 2 × h + m 3 × c 3 ) unlike the state-of-the-art algorithm solving the same problem with the complexity of O(m 3 × c 3 × h 3 ). We also propose an extension of the algorithm for clusters of h non-identical nodes where each node has c heterogeneous processors. We experimentally demonstrate the optimality of our algorithm using two well-known and highly optimized multi-threaded data-parallel applications, matrix-matrix multiplication and 2D fast Fourier transform, on a heterogeneous multi-accelerator NUMA node containing an Intel multicore Haswell CPU, an Nvidia K40c GPU, and an Intel Xeon Phi co-processor and a simulated homogeneous cluster of such nodes.

A Proposed Data Partitioning Approach on Heterogeneous HPC Platforms: Data Locality Perspective

“…Most of the previous works do not address the problem of minimizing communication cost, which is an important related concept. Modern HPC systems have become more parallel and heterogeneous to accommodate the growing demands of performance and energy efficiency [25]. The programming community is facing a great challenge in respect of how to minimize data movement, which is the most dominant factor in performance and energy consumption [6], [8], [26].…”

A Case For Intra-rack Resource Disaggregation in HPC

“…Further work has demonstrated reconigurable accelerators that rely on ield programmable gate arrays (FPGAs) [40,70] or ASICs [81]. Consequently, past work has examined how job scheduling should consider heterogeneous resource requests [8,30], how the operating system (OS) and runtime should adapt [42,57], how to write applications for heterogeneous systems [8,32], how to partition data-parallel applications onto heterogeneous compute resources [48], how to consider the diferent fault tolerances of heterogeneous resources [41], how to fairly compare the performance of diferent heterogeneous systems [44], and what the impact of heterogeneous resources is to application performance [52,74,80].…”

A Cross-Disciplinary Outlook of Directions and Challenges in Industrial Electronics

“…The schedulers dynamically distribute the processing tasks among all the resources to fully exploit all computing devices while minimizing load unbalance [67]. Amdahl's Law aims at maximizing power-normalized performance by considering the dynamic workload variations and its classification [68], [69], deploying data partitioning algorithms to minimize the parallel execution time of data-parallel applications on clusters of identical nodes of heterogeneous processors [70] or scheduling strategies and task decomposition to accelerate the training process of large-scale CNNs by minimizing the waiting time for critical paths [71].…”