LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications

Xin, Jinhan; Hwang, Kai; Yu, Zhibin

doi:10.1145/3514221.3526157

Cited by 7 publications

(9 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Concretely, they collect training samples by running jobs with different configurations on a non-production cluster, and then train a performance model to suggest new configurations for Spark job running on the production cluster. Training such an accurate performance model involves lots of offline job executions (e.g., 1000-10000) [76]. This process is very time-consuming and expensive, and often incurs data security issues when accessing business data during multiple job executions in the non-production cluster.…”

Section: Methodsmentioning

confidence: 99%

“…(C.1 Limited Functionality) Lots of methods (e.g., DAC [79], LOCAT [76]) are designed to minimize the execution time, i.e., finding the fastest configuration. However, the goal in many scenarios involves the execution cost [2,64] (i.e., the cheapest configuration), or more generalized objectives such as a weighted combination between runtime and cost.…”

Section: Methodsmentioning

confidence: 99%

“…To obtain the sub-space, we should first measure the importance of each parameter. While existing methods only consider the influence between each parameter and performance via Spearman Correlation Coefficient or weights of a learning model [24,76], we consider the importance of both single parameters and of interactions between parameters. To this end, we adopt FANOVA [35] to assess the importance of parameters.…”

Section: Sub-space Generationmentioning

confidence: 99%

“…Evolution Strategy of Sub-space. Different from existing methods that use a fixed sub-space [76], we propose to automatically adjust the size of the sub-space, i.e., 𝐾. At the beginning of the tuning process, we initialize the size of sub-space 𝐾 to a small constant 𝐾 ← 𝐾 𝑖𝑛𝑖𝑡 , and then the value of 𝐾 will be updated in a certain period, i.e., 𝑁 𝑠𝑝𝑎𝑐𝑒 iterations.…”

Section: Sub-space Generationmentioning

confidence: 99%

“…( 5) Tuneful [24] utilizes BO to tune the configuration of in-memory cluster computing systems in an online manner. ( 6) LOCAT [76] is also a BO-based online approach to conduct auto-tuning for Spark SQL. Objectives.…”

Section: Experiments Setupmentioning

confidence: 99%

See 4 more Smart Citations

Towards General and Efficient Online Tuning for Spark

Li,

Jiang,

Shen

et al. 2023

Proc. VLDB Endow.

View full text Add to dashboard Cite

The distributed data analytic system - Spark is a common choice for processing massive volumes of heterogeneous data, while it is challenging to tune its parameters to achieve high performance. Recent studies try to employ auto-tuning techniques to solve this problem but suffer from three issues: limited functionality, high overhead, and inefficient search. In this paper, we present a general and efficient Spark tuning framework that can deal with the three issues simultaneously. First, we introduce a generalized tuning formulation, which can support multiple tuning goals and constraints conveniently, and a Bayesian optimization (BO) based solution to solve this generalized optimization problem. Second, to avoid high overhead from additional offline evaluations in existing methods, we propose to tune parameters along with the actual periodic executions of each job (i.e., online evaluations). To ensure safety during online job executions, we design a safe configuration acquisition method that models the safe region. Finally, three innovative techniques are leveraged to further accelerate the search process: adaptive sub-space generation, approximate gradient descent, and meta-learning method. We have implemented this framework as an independent cloud service, and applied it to the data platform in Tencent. The empirical results on both public benchmarks and large-scale production tasks demonstrate its superiority in terms of practicality, generality, and efficiency. Notably, this service saves an average of 57.00% memory cost and 34.93% CPU cost on 25K in-production tasks within 20 iterations, respectively.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Sub-space Generationmentioning

confidence: 99%

Section: Sub-space Generationmentioning

confidence: 99%

Section: Experiments Setupmentioning

confidence: 99%

See 3 more Smart Citations

Towards General and Efficient Online Tuning for Spark

Li,

Jiang,

Shen

et al. 2023

Proc. VLDB Endow.

View full text Add to dashboard Cite

show abstract

QHB⁺: Accelerated Configuration Optimization for Automated Performance Tuning of Spark SQL Applications

Jang,

Yoon,

Jung

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Apache Spark stands out as a well-known solution for big data processing because of its efficiency and rapid processing capabilities. One of its modules, Spark SQL, serves as a prominent big data query engine. However, executing Spark SQL applications with massive data can be time-intensive, and the execution time can vary significantly depending on its configurations. Recent studies try to reduce application execution times by searching optimal configurations for applications. While Bayesian optimization is recognized as a powerful method in recent studies for configuration optimization, it faces challenges such as computational costs and time-consuming computations, especially when dealing with large search spaces Due to these challenges, we propose QHB + , designed to rapidly search optimal configurations. QHB + utilizes the Successive Halving Algorithm-based optimization methods, performing well in hyperparameter optimization of machine learning models, for configuration optimization of Spark SQL applications. Through empirical evaluations against established benchmarks, we show the efficiency of QHB + , highlighting them as swift alternatives to conventional optimization method for optimizing Spark SQL configurations.

show abstract

Prediction-driven resource provisioning for serverless container runtimes

Tomaras,

Tsenos,

Kalogeraki

2023

2023 IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS)

View full text Add to dashboard Cite

LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications

Cited by 7 publications

References 31 publications

Towards General and Efficient Online Tuning for Spark

Towards General and Efficient Online Tuning for Spark

QHB⁺: Accelerated Configuration Optimization for Automated Performance Tuning of Spark SQL Applications

Prediction-driven resource provisioning for serverless container runtimes

Contact Info

Product

Resources

About

LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications

Cited by 7 publications

References 31 publications

Towards General and Efficient Online Tuning for Spark

Towards General and Efficient Online Tuning for Spark

QHB+: Accelerated Configuration Optimization for Automated Performance Tuning of Spark SQL Applications

Prediction-driven resource provisioning for serverless container runtimes

Contact Info

Product

Resources

About

QHB⁺: Accelerated Configuration Optimization for Automated Performance Tuning of Spark SQL Applications