Real Time Autotuning for MapReduce on Hadoop/YARN

Pospelova, Maria

doi:10.22215/etd/2015-11026

Cited by 4 publications

(9 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While Hadoop and Spark have hundreds of parameters, only a few dozen of them play a significant role in determining framework performances [5] [6]. Since each parameter has a number of possible values, there are numerous combinations of these parameter values.…”

Section: Motivation and Problem Statementmentioning

confidence: 99%

“…Machine learning algorithms and Welch's t-test are applied to select the most sensitive metric in classifying the workload type and detecting workloads transitions, respectively. 6. Section 7 explains a possible online automatic tuning system inspired by the conclusions from the results of our research.…”

Section: Thesis Contributionsmentioning

confidence: 99%

“…According to past research [5] [6], the values of parameters determine the performance of analytics frameworks. For Hadoop, it has hundreds of parameters, but only dozens of them significantly affect the performance of Hadoop.…”

Section: Parameters Tuningmentioning

confidence: 99%

“…The framework is started and the application is run. 6. Steps 1 to 5 are repeated until a satisfied performance is achieved.…”

Section: Offline Auto-tuningmentioning

confidence: 99%

“…Each container executes one task based on the User-defined code. 6. AM reports the application execution status to Client.…”

Section: Introduction To Hadoop and Yarnmentioning

confidence: 99%

See 4 more Smart Citations

Improving Automatic Tuning of Hadoop and Spark by Analysing Container Performance Metrics

Zhou¹

View full text Add to dashboard Cite

This thesis attempts to provide further research on improving automatic tuning of Hadoop and Spark by analysing container performance metrics. Analytics frameworks, for example, Hadoop and Spark, are powerful tools in the study of big data. The parameter values in these analytics frameworks significantly affect the performance of applications. However, it is difficult to select the optimal framework parameter values for every application. Many research teams have proposed methods for the automatic tuning of Hadoop or Spark parameters from different perspectives. Most of the currently existing automatic tuning systems do not support working with different kinds of frameworks and Resource Managers. In addition, research on selecting the best timing to evoke the tuning method is lacking, which is another gap in the study of big data frameworks. Our research aims to fill these gaps. This research introduces novel container performance metrics and proves that these metrics are beneficial in the development of automatic tuning systems. Hadoop and Spark show different patterns in the static and dynamic values of container creation rate, container completion rate, container average response time and relative standard deviation of response-time(RSD). By applying five kinds of machine learning algorithms, container creation rate was found to be the most sensitive metric to identify and classify the workload type at an average accuracy of 83%. RSD can be used to detect workload transitions with an average accuracy of 74%. Our research results will decrease tuning overhead and promote the development of automatic tuning systems.

show abstract

Section: Motivation and Problem Statementmentioning

confidence: 99%

Section: Thesis Contributionsmentioning

confidence: 99%

Section: Parameters Tuningmentioning

confidence: 99%

“…The framework is started and the application is run. 6. Steps 1 to 5 are repeated until a satisfied performance is achieved.…”

Section: Offline Auto-tuningmentioning

confidence: 99%

“…Each container executes one task based on the User-defined code. 6. AM reports the application execution status to Client.…”

Section: Introduction To Hadoop and Yarnmentioning

confidence: 99%

See 3 more Smart Citations

Improving Automatic Tuning of Hadoop and Spark by Analysing Container Performance Metrics

Zhou¹

View full text Add to dashboard Cite

show abstract

Time and Cost Prediction Models for Language Classification Over a Large Corpus on Spark

Rodrigues

Vasconcelos

Maciel

2020

2020 IEEE Symposium Series on Computational Intelligence (SSCI)

View full text Add to dashboard Cite

Improving Real Time Tuning on YARN

Belmonte¹,

Pablo²

View full text Add to dashboard Cite

Big data is becoming a significant part of the operations of modern organizations. Performing analysis of large amounts of data requires computer clusters to run the calculations and analysis. YARN is an internal framework that is responsible for coordinating big data jobs for some popular distributed storage and processing frameworks like Hadoop and Spark. Running YARN with the correct configuration parameters is critical for the good performance of a cluster. KERMIT is an online tuner of YARN configuration parameters that aims to improve cluster performance. The first KER-MIT implementation proved the feasibility of the concept. In this study we modified the tuning algorithms inside the KERMIT components; by doing so, we achieved a reduction in the execution time as compared to industry benchmarks for a shallow tuning technique. We also verified that KERMIT can tune Spark; this suggests that Kermit could be used in other YARN-based frameworks.

show abstract

Real Time Autotuning for MapReduce on Hadoop/YARN

Cited by 4 publications

References 22 publications

Improving Automatic Tuning of Hadoop and Spark by Analysing Container Performance Metrics

Improving Automatic Tuning of Hadoop and Spark by Analysing Container Performance Metrics

Time and Cost Prediction Models for Language Classification Over a Large Corpus on Spark

Improving Real Time Tuning on YARN

Contact Info

Product

Resources

About