Practical aggregation of semantical program properties for machine learning based optimization

Abstract

Iterative search combined with machine learning is a promising approach to design optimizing compilers harnessing the complexity of modern computing systems. While traversing a program optimization space, we collect characteristic feature vectors of the program, and use them to discover correlations across programs, target architectures, data sets, and performance. Predictive models can be derived from such correlations, effectively hiding the time-consuming feedback-directed optimization process from the application programmer.One key task of this approach, naturally assigned to compiler experts, is to design relevant features and implement scalable feature extractors, including statistical models that filter the most relevant information from millions of lines of code. This new task turns out to be a very challenging and tedious one from a compiler construction perspective. So far, only a limited set of ad-hoc, largely syntactical features have been devised. Yet machine learning is only able to discover correlations from information it is fed with: it is critical to select topical program features for a given optimization problem in order for this approach to succeed.We propose a general method for systematically generating numerical features from a program. This method puts no restrictions on how to logically and algebraically aggregate semantical properties into numerical features. We illustrate our method on the difficult problem of selecting the best possible combination of 88 available optimizations in GCC. We achieve 74% of the potential speedup obtained through iterative compilation on a wide range of benchmarks and four different general-purpose and embedded architectures. Our work is particularly relevant to embedded system designers willing to quickly adapt the optimization heuristics of a mainstream compiler to their custom ISA, microarchitecture, benchmark suite and workload. Our method has been integrated with the publicly released MILEPOST GCC [14].

Milepost GCC: Machine Learning Enabled Self-tuning Compiler

“…The extract program static features plugin invokes a Prolog compiler to execute this program, resulting in a vector of features (as shown in Table 2) which later serves to detect similarities between programs, build machine learning models and predict the best combinations of passes for new programs. We provide more details about aggregation of semantical program properties for machine learning based optimization in [68].…”

End-to-End Deep Learning of Optimization Heuristics

“…A few methods have been proposed to automatically generate features from the compiler's intermediate representation [8,9]. These approaches closely tie the implementation of the predictive model to the compiler IR, which means changes to the IR will require modifications to the model.…”

“…Although much of this can be created from templates, selecting the right range of capabilities and search space bias is non trivial and up to the expert. The work of [8] expresses the space of features via logic programming over relations that represent information from the IRs. It greedily searches for expressions that represent good features.…”

“…While techniques for automatic feature generation from the compiler IR have been proposed in the past [8,9], they do not solve the problem in a practical way. They are deeply embedded into the compiler, require expert knowledge to guide the generation, have to be repeated from scratch for every new heuristic, and their search time can be prohibitive.…”

Predictive Modeling in a Polyhedral Optimization Space

“…Although significant performance improvements have been demonstrated [26], [1], [17], [28], the performance obtained has generally been limited by the optimizations selected for automatic tuning, and by the degrees of freedom available for exploration. We identify two main limitations of iterative compilation efforts so far.…”