Black or White? How to Develop an AutoTuner for Memory-based Analytics

Abstract

There is a lot of interest today in building autonomous (or, self-driving) data processing systems. An emerging school of thought is to leverage AI-driven "black box" algorithms for this purpose. In this paper, we present a contrarian view. We study the problem of autotuning the memory allocation for applications running on modern distributed data processing systems. We show that an empirically-driven "white-box" algorithm, called RelM, that we have developed provides a close-to-optimal tuning at a fraction of the overheads compared to state-of-the-art AI-driven "black box" algorithms, namely, Bayesian Optimization (BO) and Deep Distributed Policy Gradient (DDPG). The main reason for RelM's superior performance is that the memory management in modern memory-based data analytics systems is an interplay of algorithms at multiple levels: (i) at the resource-management level across various containers allocated by resource managers like Kubernetes and YARN, (ii) at the container level among the OS, pods, and processes such as the Java Virtual Machine (JVM), (iii) at the application level for caching, aggregation, data shuffles, and application data structures, and (iv) at the JVM level across various pools such as the Young and Old Generation. RelM understands these interactions and uses them in building an analytical solution to autotune the memory management knobs. In another contribution, called Guided-BO (GBO), we use RelM's analytical models to speed up BO. Through an evaluation based on Apache Spark, we showcase that the RelM's recommendations are significantly better than what commonly-used Spark deployments provide, and are close to the ones obtained by brute-force * This research is supported by NSF grant IIS-1423124.

Facilitating Database Tuning with Hyper-Parameter Optimization: A Comprehensive Experimental Evaluation

“…Recently, there has been an active research area on automatically tuning database configurations using Machine Learning (ML) techniques [3,18,24,44,47,55,91,92]. We summarize three key modules in the existing tuning systems: knob selection that prunes the configuration space, configuration optimization that samples promising configurations over the pruned space, and knowledge transfer that further speeds up the tuning process via historical data.…”

A Survey on Deep Reinforcement Learning for Data Processing and Analytics

“…Kunjir et al [38] investigate memory allocation auto-tuning for applications on distributed data processing systems and propose a white-box algorithm RelM. The RelM is developed to empirically model the interactions of memory management options and provides analytical models to estimate different competing memory pools requirements in an application.…”

You Only Run Once: Spark Auto-Tuning From a Single Run

“…Auto-tuning with a performance model which leverages an application characterization to aid the search: [15] and [30] are two publications in the area of Spark auto-tuning that proposes more similar systems to our work, generalizing unseen workloads by some characterization of the application. [15] approach to extract the features is more close to our work as it capture Task and Stage information from an Spark application, while [30] follows a more general approach by profiling the application to extract statistics like the average CPU and disk usage that can work with other Big Data frameworks rather than Spark.…”

“…In [30], a vector of statistics is extracted from the runtime of the application to be optimized. This feature vector is used to modify a BO procedure to guide the search process.…”

“…Our work shares the core idea of providing specific knowledge about the running application to improve the search process. The main difference is that, in [30], the authors do not try to proxy application executions with metrics provided with an oracle.…”