Towards Automatic Tuning of Apache Spark Configuration

Nguyen, Nhan; Khan, Mohammad Maifi Hasan; Wang, Kewen

doi:10.1109/cloud.2018.00059

Cited by 25 publications

(27 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In a related, but different direction, Nguyen et al [17] proposed a strategy to generate training data to fit a performance model. The model is meant to be used for predicting which Spark settings yield the smallest application execution time (i.e., capacity planning).…”

Section: Related Workmentioning

confidence: 99%

Machine Learning for Performance Prediction of Spark Cloud Applications

Maros

Murai

Silva

et al. 2019

2019 IEEE 12th International Conference on Cloud Computing (CLOUD)

View full text Add to dashboard Cite

Big data applications and analytics are employed in many sectors for a variety of goals: improving customers satisfaction, predicting market behavior or improving processes in public health. These applications consist of complex software stacks that are often run on cloud systems. Predicting execution times is important for estimating the cost of cloud services and for effectively managing the underlying resources at runtime. Machine Learning (ML), providing black box solutions to model the relationship between application performance and system configuration without requiring in-detail knowledge of the system, has become a popular way of predicting the performance of big data applications. We investigate the cost-benefits of using supervised ML models for predicting the performance of applications on Spark, one of today's most widely used frameworks for big data analysis. We compare our approach with Ernest (an ML-based technique proposed in the literature by the Spark inventors) on a range of scenarios, application workloads, and cloud system configurations. Our experiments show that Ernest can accurately estimate the performance of very regular applications, but it fails when applications exhibit more irregular patterns and/or when extrapolating on bigger data set sizes. Results show that our models match or exceed Ernest's performance, sometimes enabling us to reduce the prediction error from 126-187% to only 5-19%.

show abstract

Section: Related Workmentioning

confidence: 99%

Machine Learning for Performance Prediction of Spark Cloud Applications

Maros

Murai

Silva

et al. 2019

2019 IEEE 12th International Conference on Cloud Computing (CLOUD)

View full text Add to dashboard Cite

show abstract

“…They then focus on optimizing the subset of relevant parameters, considerably reducing the number of samples needed to achieve good results. In order to avoid running an application many times to find a good configuration, many works use a machine learning model trained to predict the evaluation metric from past executions [8], [11]- [15]. This model, sometimes called the performance model, aims to replace many of the job execution calls with the predictions provided by the model.…”

Section: A Popular Methodologies For the Problemmentioning

confidence: 99%

“…Note that in equation (2) ∈ can change but ∈ is fixed. In some works [7], [8], is a single variable containing the dataset size. This implies that the only information expected to affect the performance of an application is the dataset size.…”

Section: Problem Formulation and Motivationmentioning

confidence: 99%

You Only Run Once: Spark Auto-Tuning From a Single Run

Buchaca

Portella

Costa

et al. 2020

IEEE Trans. Netw. Serv. Manage.

View full text Add to dashboard Cite

Tuning configurations of Spark jobs is not a trivial task. State-of-the-art auto-tuning systems are based on iteratively running workloads with different configurations. During the optimization process, the relevant features are explored to find good solutions. Many optimizers enhance the time-to-solution using black-box optimization algorithms that do not take into account any information from the Spark workloads. In this paper, we present a new method for tuning configurations that uses information from one run of a Spark workload. To achieve good performance, we mine the SparkEventLog that is generated by the Spark engine. This log file contains a large amount of information from the executed application. We use this information to enhance a performance model with low-level features from the workload to be optimized. These features include Spark Actions, Transformations, and Task metrics. This process allows us to obtain application-specific workload information. With this information our system can predict sensible Spark configurations for unseen jobs, given that it has been trained with reasonable coverage of Spark applications. Experiments show that the presented system correctly produces good configurations, while achieving up to 80% speedup with respect to the default Spark configuration, and up to 12x speedup of the time-to-solution with respect to a standard Bayesian Optimization procedure.

show abstract

“…Recently, research on tuning Spark performance has been on the rise [11,25,31,40]. This is due to the increase in the number of applications that use Spark for data intensive tasks [13,28,29]. The work described in [29] is an attempt to find the parameters to tune Spark performance using ML.…”

Section: Related Workmentioning

confidence: 99%

“…This is due to the increase in the number of applications that use Spark for data intensive tasks [13,28,29]. The work described in [29] is an attempt to find the parameters to tune Spark performance using ML. Identifying the best Spark configuration parameters for a specific application is very challenging.…”

Section: Related Workmentioning

confidence: 99%

Exploiting Spark for HPC Simulation Data

Jiang

Gallagher

Chu

et al. 2020

Proceedings of the International Conference on High Performance Computing in Asia-Pacific Region

View full text Add to dashboard Cite

In this paper, we address the challenge of analyzing simulation data on HPC systems by using Apache Spark, which is a Big Data framework. One of the main problems we encountered with using Spark on HPC systems is the ephemeral data explosion, which is brought about by the curse of persistence in the Spark framework. Data persistence is essential in reducing I/O, but it comes at the cost of storage space. We show that in some cases, Spark scratch data can consume an order of magnitude more space than the input data being analyzed, leading to fatal out-of-disk errors. We investigate the real-world application of scaling machine learning algorithms to predict and analyze failures in multi-physics simulations on 76TB of data (over one trillion training examples). This problem is 2-3 orders of magnitude larger than prior work. Based on extensive experiments at scale, we provide several concrete recommendations as state-of-the-practice, and demonstrate a 7x reduction in disk utilization with negligible increases or even decreases in runtime.

show abstract

Towards Automatic Tuning of Apache Spark Configuration

Cited by 25 publications

References 11 publications

Machine Learning for Performance Prediction of Spark Cloud Applications

Machine Learning for Performance Prediction of Spark Cloud Applications

You Only Run Once: Spark Auto-Tuning From a Single Run

Exploiting Spark for HPC Simulation Data

Contact Info

Product

Resources

About