Machine learning for predictive auto-tuning with boosted regression trees

Abstract

The rapidly evolving landscape of multicore architectures makes the construction of efficient libraries a daunting task. A family of methods known collectively as "auto-tuning" has emerged to address this challenge. Two major approaches to auto-tuning are empirical and model-based: empirical autotuning is a generic but slow approach that works by measuring runtimes of candidate implementations, model-based auto-tuning predicts those runtimes using simplified abstractions designed by hand. We show that machine learning methods for non-linear regression can be used to estimate timing models from data, capturing the best of both approaches. A statistically-derived model offers the speed of a model-based approach, with the generality and simplicity of empirical auto-tuning. We validate our approach using the filterbank correlation kernel described in Pinto and Cox [2012], where we find that 0.1 seconds of hill climbing on the regression model ("predictive auto-tuning") can achieve almost the same speed-up as is brought by minutes of empirical auto-tuning. Our approach is not specific to filterbank correlation, nor even to GPU kernel auto-tuning, and can be applied to almost any templated-code optimization problem, spanning a wide variety of problem types, kernel types, and platforms.

Machine learning‐based auto‐tuning for enhanced performance portability of OpenCL applications

“…The works by Bergstra et al, Jia et al, and Nugteren et al are those most closely related to ours. In Bergstra et al, a model based on boosted regression trees was used to build an auto‐tuner, evaluated with a single GPU benchmark, filterbank correlation.…”

“…The works by Bergstra et al, Jia et al, and Nugteren et al are those most closely related to ours. In Bergstra et al, a model based on boosted regression trees was used to build an auto‐tuner, evaluated with a single GPU benchmark, filterbank correlation. The Starchart system was used to build a regression tree model that could be used to partition the design space of an application, discover its structure, and find optimal parameter values within the different regions.…”

Performance‐influence models of multigrid methods: A case study on triangular grids

“…For the optimal designs, we selected the D‐Optimal Design , because it selects configurations according to an optimization criteria that fits best for our purpose (configurations with the largest distance to each other are selected). Last, we also use a random selection of variants, because this is a standard method in machine learning and often leads to good results …”

Genesis

“…There is increasing interest in the use of machine learning in automatic tuning of program performance, particularly on parallel architectures such as multi-cores and Graphics Processing Units (GPUs) [1,5,8,9,16,19]. Much of this work uses supervised learning methods that rely on training programs, i.e., programs to which various optimizations are applied and whose characteristics and performance are used to build a machine learning model.…”