Automatic, On-Line Tuning of YARN Container Memory and CPU Parameters

Genkin, Mikhail; Dehne, Frank; Pospelova, Maria; Ya-bing, Chen; Navarro, Pablo

doi:10.1109/hpcc-smartcity-dss.2016.0053

Cited by 17 publications

(35 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The offline auto-tuning system needs to restart the cluster to modify the values of parameters. Based on the research paper, "Mronline" [8], the process of offline autotuning is summarised as follows: 1. Several tests are running with initial parameter values and modified values.…”

Section: Offline Auto-tuningmentioning

confidence: 99%

“…Kermit is the first on-line automatic tuning system for YARN [1]. Kermit was released before I joined the team.…”

Section: Overview Of Kermitmentioning

confidence: 99%

“…Kermit was released before I joined the team. The published paper "Automatic, on-line tuning of YARN container memory and CPU parameters" [1] provides more details about Kermit.…”

Section: Overview Of Kermitmentioning

confidence: 99%

“…Based on this paper [1], the design of Kermit, and inspiration from Kermit will be introduced in this Section.…”

Section: Overview Of Kermitmentioning

confidence: 99%

“…Since JobTacker is in single mode and is multi-purpose, it decreases the scalability of Hadoop. Hadoop Figure 2.1, the process of Hadoop application execution [11] is summarised as follows: 1. Client submits the application to the master node.…”

Section: Introduction To Hadoop and Yarnmentioning

confidence: 99%

See 4 more Smart Citations

Improving Automatic Tuning of Hadoop and Spark by Analysing Container Performance Metrics

Zhou¹

View full text Add to dashboard Cite

This thesis attempts to provide further research on improving automatic tuning of Hadoop and Spark by analysing container performance metrics. Analytics frameworks, for example, Hadoop and Spark, are powerful tools in the study of big data. The parameter values in these analytics frameworks significantly affect the performance of applications. However, it is difficult to select the optimal framework parameter values for every application. Many research teams have proposed methods for the automatic tuning of Hadoop or Spark parameters from different perspectives. Most of the currently existing automatic tuning systems do not support working with different kinds of frameworks and Resource Managers. In addition, research on selecting the best timing to evoke the tuning method is lacking, which is another gap in the study of big data frameworks. Our research aims to fill these gaps. This research introduces novel container performance metrics and proves that these metrics are beneficial in the development of automatic tuning systems. Hadoop and Spark show different patterns in the static and dynamic values of container creation rate, container completion rate, container average response time and relative standard deviation of response-time(RSD). By applying five kinds of machine learning algorithms, container creation rate was found to be the most sensitive metric to identify and classify the workload type at an average accuracy of 83%. RSD can be used to detect workload transitions with an average accuracy of 74%. Our research results will decrease tuning overhead and promote the development of automatic tuning systems.

show abstract

Section: Offline Auto-tuningmentioning

confidence: 99%

“…Kermit is the first on-line automatic tuning system for YARN [1]. Kermit was released before I joined the team.…”

Section: Overview Of Kermitmentioning

confidence: 99%

“…Kermit was released before I joined the team. The published paper "Automatic, on-line tuning of YARN container memory and CPU parameters" [1] provides more details about Kermit.…”

Section: Overview Of Kermitmentioning

confidence: 99%

“…Based on this paper [1], the design of Kermit, and inspiration from Kermit will be introduced in this Section.…”

Section: Overview Of Kermitmentioning

confidence: 99%

Section: Introduction To Hadoop and Yarnmentioning

confidence: 99%

See 3 more Smart Citations