Performance modeling of heterogeneous systems

Abstract

As the complexity of parallel computers grows, constraints posed by the construction of larger systems require both greater, and increasingly non-linear, parameter sets to model their behavior realistically. These heterogeneous characteristics create a trade-off between the complexity and accuracy of performance models, creating challenges in utilizing them for design decisions.In this thesis, we take a bottom-up approach to realistically model software and hardware interactions, by composing system models from simpler, linear models, which allow parts of the analysis to be automated. We associate empirically benchmarked platform performance metrics with the core elements in a variant of bulk-synchronous execution, aiming to quantify application performance, and associated potential for computation and communication overlap on SMP clusters.The original bulk-synchronous performance model is introduced, and we identify areas of computation and communication where its abstractions impede realistic models of contemporary hardware. These are addressed independently, using experimental evidence to develop a representation collecting computation kernel characteristics and pairwise communications in matrices, to combine into a system model. As bulk-synchronous execution strongly depends on periodic, global synchronization, we develop a cost model for it by combining latency measurements with a parametric representation of signalling patterns, and experimentally verify the resulting predictions for three common algorithms.We describe a design to implement the BSPLib programming interface, combining threads and message-passing parallelism to achieve overlap on commodity cluster platforms, implementing its one-sided communication primitives using out-of-band control messages. We augment and validate the cost model of one adapted synchronization algorithm with the corresponding bandwidth requirement, completing a framework for modeling BSPLib program performance.Finally, we test the utility of this framework as a proof-of-concept for guiding software performance adaptations, using two cases. First, we use the latency terms to automatically generate synchronization operations, using model predictions to generate customized patterns with respect to platform topology, showing that the resulting algorithms equal or outperform the system defaults. Second, the strong scaling characteristics of a 5-point stencil code is compared for three implementations. Experiments show the performance overhead of our implementation, but also its capability for predicting program cost, including parameter values to optimize for balanced overlapping of computation and communication.ii Preface

ImageCL: Language and source‐to‐source compiler for performance portability, load balancing, and scalability prediction on heterogeneous systems

“…Much work have also been done on performance modeling for heterogeneous systems,() including approaches using machine learning and performance counters. Wu et al predicted how kernels scale when GPU hardware properties change, based on performance counter values, using a machine learning model.…”

“…In the following, we will simplify the analysis by restricting ourselves to ImaceCL‐like applications, as described in Section 6. These applications follow the bulk synchronous parallel (BSP) model,() consisting of stages of parallel computation on local data, interleaved with global communication and synchronization. Our method is, however, not tied to ImageCL and can also be generalized to other models than BSP.…”

Machine learning‐based auto‐tuning for enhanced performance portability of OpenCL applications

“…Furthermore, analytical performance models for GPUs and heterogeneous systems have been developed [42][43][44][45] and used for auto-tuning. 5 Much work has been done on creating machine learning-based models and using them for tuning and auto-tuning, eg, to determine loop unroll factors, 28,46 which optimizations to apply for parallel stencil computations, 7,47 Message Passing Interface (MPI) parameters, 14 and general compiler optimizations.…”

“…Khan et al used a script‐based auto‐tuning compiler to translate sequential C loop nests to parallel CUDA code. Furthermore, analytical performance models for GPUs and heterogeneous systems have been developed and used for auto‐tuning …”

Machine Learning Based Auto-Tuning for Enhanced OpenCL Performance Portability

“…for stencil computations [7], matrix multiplication [8] and FFTs [9]. Furthermore, analytical performance models for GPUs and heterogeneous systems have been developed [10,11,12,13] and used for auto-tuning [14].…”