Celebrating diversity: a mixture of experts approach for runtime mapping in dynamic environments

Abstract

Matching program parallelism to platform parallelism using thread selection is difficult when the environment and available resources dynamically change. Existing compiler or runtime approaches are typically based on a one-size fits all policy. There is little ability to either evaluate or adapt the policy when encountering new external workloads or hardware resources. This paper focuses on selecting the best number of threads for a parallel application in dynamic environments. It develops a new scheme based on a mixture of experts approach. It learns online which, of a number of existing policies, or experts, is best suited for a particular environment without having to try out each policy. It does this by using a novel environment predictor as a proxy for the quality of an expert thread selection policy. Additional expert policies can easily be added and are selected only when appropriate. We evaluate our scheme in environments with varying external workloads and hardware resources. We then consider the case when workloads use affinity scheduling or are themselves adaptive and show that our approach, in all cases, outperforms existing schemes and surprisingly improves workload performance. On average, we improve 1.66x over OpenMP default, 1.34x over an online scheme, 1.25x over an offline policy and 1.2x over a state-of-art analytic model. Determining the right number and type of experts is an open problem and our initial analysis shows that adding more experts improves accuracy and performance.

Machine Learning in Compiler Optimization

“…This is because a one-size-fits-all model is unlikely to precisely capture behaviours of diverse applications, and no matter how parameterized the model is, it is highly unlikely that a model developed today will always be suited for tomorrow. To allow the model to adapt to the change of the computing environment and workloads, ensemble learning was exploited in prior works [73], [119], [120]. The idea of ensemble learning is to use multiple learning algorithms, where each algorithm is effective for particular problems, to obtain better predictive performance than could be obtained from any of the constituent learning algorithm alone [121], [122].…”

“…Various forms of program features have been used in compiler-based machine learning. These include static code structures [123] and runtime information such as system load [119], [124] and performance counters [53].…”

“…In [174] and [175], an ANN model based on static code features and runtime information was built to predict the number of threads to use for a target program when it runs with external workloads. Later in [119] an ensemble learning based approach was used, which leads to significantly better performance over [174]. In [119] several models are firstly trained offline; and then one of the model is selected at runtime, taking into consideration the competing workloads and available hardware resources.…”

“…Later in [119] an ensemble learning based approach was used, which leads to significantly better performance over [174]. In [119] several models are firstly trained offline; and then one of the model is selected at runtime, taking into consideration the competing workloads and available hardware resources. The central idea is that instead of using a single monolithic model, we can use multiple models where each model is specialised for mode ling a subset of applications or a particular runtime scenario.…”

Scalo

“…Existing approaches are limited in accuracy and usability. Offline profiling approaches [12,18] can miss contention effects that arise during co-execution. Moreover, they require extensive and lengthy runs with different configurations to build scalability models.…”