Fast Automatic Heuristic Construction Using Active Learning

Abstract

Abstract. Building effective optimization heuristics is a challenging task which often takes developers several months if not years to complete. Predictive modelling has recently emerged as a promising solution, automatically constructing heuristics from training data. However, obtaining this data can take months per platform. This is becoming an ever more critical problem and if no solution is found we shall be left with out of date heuristics which cannot extract the best performance from modern machines. In this work, we present a low-cost predictive modelling approach for automatic heuristic construction which significantly reduces this training overhead. Typically in supervised learning the training instances are randomly selected to evaluate regardless of how much useful information they carry. This wastes effort on parts of the space that contribute little to the quality of the produced heuristic. Our approach, on the other hand, uses active learning to select and only focus on the most useful training examples. We demonstrate this technique by automatically constructing a model to determine on which device to execute four parallel programs at differing problem dimensions for a representative Cpu-Gpu based heterogeneous system. Our methodology is remarkably simple and yet effective, making it a strong candidate for wide adoption. At high levels of classification accuracy the average learning speed-up is 3x, as compared to the stateof-the-art.

Using meta-heuristics and machine learning for software optimization of parallel computing systems: a systematic literature review

“…With regards to static scheduling, the attention of recent research that use machine learning and meta-heuristics is in the following optimization objectives: mapping program parallelism to multi-core architectures [98], mapping applications to the most appropriate processing device [40,71], mapping threads to specific cores [15], and determining workload distribution on heterogeneous parallel computing systems [62][63][64][65].…”

“…An approach that combines a number of machine learning algorithms, including, Logistic (L), Multilayer Perceptron (MP), IB1, IBk, KStar, Random Forest, Logit Boost, Multi-Class-Classifier, Random Committee, NNge, ADTree, and RandomTree, to create an active-learning query-committee with the aim to reduce the required amount if training data is proposed by Ogilvie et al [71]. A combination of Simulated Annealing (SA) and boosted decision tree regression to determine near optimal system configurations is proposed by Memeti and Pllana [63].…”

“…Grewe et al [41] adapts to the workload only when the application starts its execution, but it does not adapt throughout its execution. Zhang et al [102] focus on applications that consist of single-loops [15] Determining thread mapping strategy for TM applications Since it uses static features, it can not change the mapping strategy when the parallelism degree changes at runtime [56] Determining near-optimal system configuration parameters Requires extensive profiling data collection and analysis, which may introduce significant run-time overhead [24,74,75] Dynamic task scheduling The proposed approaches do not consider dependencies between tasks [80] Dividing tasks into chunks and scheduling in task-farm way Task-farm or master-slave like scheduling techniques requires no profiling, however they introduce communication overheads [2,104,105] Task scheduling Assume that communication time is known prior execution, processing units have equal computing power and are always ready to perform tasks, and scheduling can be determined off-line and it can not be changed at run-time [71] Reducing the number of required training data This solution requires additional programming investment to build the prediction committee [4] Mapping of computation to heterogeneous systems Collecting profiling information for each kernel on every device represent huge overhead, especially on heterogeneous systems that may comprise larger number of non-identical CPUs and/or GPUs [91] Adaptive algorithm selection Considers hardware characteristics, but not the program input characteristics. Some algorithms perform better for smaller input sizes, whereas others perform better with larger ones [28,29,46,47,58,84] Self-adaptation Require adding additional information to the code.…”

“…Many static scheduling approaches require retraining of the prediction model for each new architecture, limiting their general use because training requires a significant amount of data that is not always available. Approaches that reduce the amount of training data require implementation of multiple machine learning algorithms (for instance, [71]). Approaches that use a single execution [21,56] by trying various system configurations during the program execution are promising, however the introduced overhead is not negligible.…”

Minimizing the cost of iterative compilation with active learning

“…Researchers have improved upon this work by removing its reliance on random search and used active learning instead [4,5,39,60]. Random search is problematic because it selects optimization decisions and profiles the application multiple times under those optimizations before it even knows whether this will actually improve our knowledge of the decision space.…”