Finding Faster Configurations Using FLASH

Abstract

Finding good configurations of a software system is often challenging since the number of configuration options can be large. Software engineers often make poor choices about configuration or, even worse, they usually use a sub-optimal configuration in production, which leads to inadequate performance. To assist engineers in finding the better configuration, this article introduces FLASH, a sequential model-based method that sequentially explores the configuration space by reflecting on the configurations evaluated so far to determine the next best configuration to explore. FLASH scales up to software systems that defeat the prior state-of-the-art model-based methods in this area. FLASH runs much faster than existing methods and can solve both single-objective and multi-objective optimization problems. The central insight of this article is to use the prior knowledge of the configuration space (gained from prior runs) to choose the next promising configuration. This strategy reduces the effort (i.e., number of measurements) required to find the better configuration. We evaluate FLASH using 30 scenarios based on 7 software systems to demonstrate that FLASH saves effort in 100% and 80% of cases in single-objective and multi-objective problems respectively by up to several orders of magnitude compared to state-of-the-art techniques.

Whence to Learn? Transferring Knowledge in Configurable Systems Using BEETLE

“…A typical approach uses sensitivity analysis [29], where performance models are learned by measuring the performance of the system under a limited number of sampled configurations. While this approach is cheaper and more effective than manual exploration, it still incurs the expense of extensive data collection about the software [3], [4], [7], [8], [10], [12], [26], [27], [30]. This is undesirable since this data collection has to be repeated if ever the software is updated or the environment of the system changes abruptly.…”

“…It has recently been shown that exact measures like MMRE can be somewhat misleading to assess configurations [12], [27], [36].…”

“…An alternative is to use rank-based metrics that compute the difference between the relative rankings of the performance scores [12], [27]. The key intuition behind relative rankings is that the raw accuracy (as measured by MMRE) is less important than the rank-ordering of configurations from best to worst.…”

“…As evidenced by our results that are remarkably stable over 30 repeated runs. • Learner Bias: There are various models used in performance optimization such as Gaussian Process [25], Regression Trees [7], [26], [27], and Bagging, Random Forest, and Support Vector Machines (SVMs) [5]. It is possible that changing the learner used may change our findings.…”

“…It is possible that changing the learner used may change our findings. However, we strive to minimize the uncertainty by choosing Decision Tree Regressor, which is the machine learning algorithm that has most consistently been used in the domain of performance modeling and optimization [7], [26], [27]. Further, we have made available our replication package that enables one to replace Decision Tree with any other machine learning model quickly.…”