Facilitating Database Tuning with Hyper-Parameter Optimization: A Comprehensive Experimental Evaluation

Abstract

Recently, using automatic configuration tuning to improve the performance of modern database management systems (DBMSs) has attracted increasing interest from the database community. This is embodied with a number of systems featuring advanced tuning capabilities being developed. However, it remains a challenge to select the best solution for database configuration tuning, considering the large body of algorithm choices. In addition, beyond the applications on database systems, we could find more potential algorithms designed for configuration tuning. To this end, this paper provides a comprehensive evaluation of configuration tuning techniques from a broader perspective, hoping to better benefit the database community. In particular, we summarize three key modules of database configuration tuning systems and conduct extensive ablation studies using various challenging cases. Our evaluation demonstrates that the hyper-parameter optimization algorithms can be borrowed to further enhance the database configuration tuning. Moreover, we identify the best algorithm choices for different modules. Beyond the comprehensive evaluations, we offer an efficient and unified database configuration tuning benchmark via surrogates that reduces the evaluation cost to a minimum, allowing for extensive runs and analysis of new techniques.

LlamaTune: Sample-Efficient DBMS Configuration Tuning

“…The actor utilizes a deep neural network that is trained based on DBMS internal metrics, and can propose configurations that balance exploration and exploitation. In general, in order for the RL methods to work well, up to thousands of samples have to be evaluated [17] and prior work [37] has found that RL based methods require more iterations due to the complexity of the neural networks used.…”

“…We observe that existing optimizers do not leverage expert knowledge about the system they are tuning and using domain knowledge has the potential to significantly improve sample efficiency. First, based on our prior work [15] (and other prior work [32,37]) we find that tuning a few important knobs is sufficient for achieving high DBMS performance, and tuning a smaller configuration space can lead to significant improvements in the number of samples required [37]. Yet, which knobs are important varies by workload, and existing methods for identifying important knobs are expensive and unreliable (Section 2.3).…”

“…As a result, a number of automated methods [3,10,13,17,31,33,36,38,39] for tuning have been proposed, with recent methods using machine learning (ML). ML-based methods have been shown to generalize to a number of workloads [37] and can find configurations that achieve up to 3-6× higher throughput (or lower latency) [17,31,36] when compared to using manually-tuned configurations.…”

“…Algorithms used in these configuration optimizers try to balance between exploring unseen regions of the configuration search space, and exploiting the knowledge gathered until that point. Search-based methods [39], RL-based methods [17] or Bayesian Optimization (BO) methods [3,10] are commonly used, though a recent study [37] found that SMAC [14], a BO-based method, performed best for DBMS tuning.…”

“…Based on our visibility into production services operating millions of databases in Azure, we find that sample efficiency (i.e., how few workload runs does it take to reach a given level of performance) is crucial to scale autotuning across diverse workloads. Currently, state-of-the-art methods require 100 or more iterations [37] to converge to a good configuration, with each iteration taking several minutes 1 .…”