End-to-End Deep Learning of Optimization Heuristics

Abstract

Accurate automatic optimization heuristics are necessary for dealing with the complexity and diversity of modern hardware and software. Machine learning is a proven technique for learning such heuristics, but its success is bound by the quality of the features used. These features must be hand crafted by developers through a combination of expert domain knowledge and trial and error. This makes the quality of the final model directly dependent on the skill and available time of the system architect.Our work introduces a better way for building heuristics. We develop a deep neural network that learns heuristics over raw code, entirely without using code features. The neural network simultaneously constructs appropriate representations of the code and learns how best to optimize, removing the need for manual feature creation. Further, we show that our neural nets can transfer learning from one optimization problem to another, improving the accuracy of new models, without the help of human experts.We compare the effectiveness of our automatically generated heuristics against ones with features hand-picked by experts. We examine two challenging tasks: predicting optimal mapping for heterogeneous parallelism and GPU thread coarsening factors. In 89% of the cases, the quality of our fully automatic heuristics matches or surpasses that of state-of-theart predictive models using hand-crafted features, providing on average 16% and 12% more performance with no human effort expended on designing features.

A benchmark set of highly-efficient CUDA and OpenCL kernels and its dynamic autotuning with Kernel Tuning Toolkit

“…In [33], a dynamic selection from a very limited number of code variants is based on a model created from previous tuning runs. In [8], a single tuning parameter can be optimized at compilation time by a neural network trained in multiple trial runs. Contrary to those papers, we focus on multi-dimensional tuning spaces.…”

“…The class inherited from tuning manipulator must override the method launchComputation, which is responsible for executing the two kernels via KTT in our example, but it could also implement computation in C++ or call KTT functions for data movement or synchronization. void launchComputation ( const k t t : : K e r n e l I d ) 8 o v e r r i d e { 9…”

Machine Learning in Compiler Optimization

“…The first work to employ neural network to extract features from program source code for compiler optimisation is 14 conducted by Cummins et al [78]. Their system, namely DeepTune, automatically abstracts and selects appropriate features from the raw source code.…”

“…Recently, it has been employed to model program source code to obtain a compact set of features that can characterise the input program source [147], [148], [149], [78], [150].…”

“…It is worth mentioning that many of these recent works show that the past development knowledge extracted from large code bases such as GitHub are valuable for learning an effective model. There were two recent studies performed by Cummins et al, which mine Github to synthesize OpenCL benchmarks [149] and code extract features from source code [78]. Both studies demonstrate the usefulness of large code bases and deep learning techniques for learning predictive models for compiler optimizations.…”