Machine-Learning Based Spark and Hadoop Workload Classification Using Container Performance Patterns

Genkin, Mikhail; Dehne, Frank; Navarro, Pablo; Zhou, Siyu

doi:10.1007/978-3-030-32813-9_11

Cited by 9 publications

(20 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based on this section, the readers will understand how parameter tuning works in the analytics frameworks. 2. Section 3 reviews the state of the art of tuning analytics frameworks and features of workloads.…”

Section: Thesis Contributionsmentioning

confidence: 99%

“…Several tests are running with initial parameter values and modified values. 2. Job performance and system resource usage logs are collected.…”

Section: Offline Auto-tuningmentioning

confidence: 99%

“…Since there are many analytics frameworks in use, it is necessary to have a tuning system that can apply to several frameworks. 2. The data size of these previous research experiments is on the GB level which is far from scenarios faced in industry.…”

Section: Limitations Of Previous Researchmentioning

confidence: 99%

“…1. Choose an appropriate tuning timing by monitoring workload transitions 2. According to the current workload type, select a specific tuning strategy.…”

Section: Sectionmentioning

confidence: 99%

“…Client submits the application to the master node. 2. RM launches application master (AM) in the worker nodes.…”

Section: Introduction To Hadoop and Yarnmentioning

confidence: 99%

See 4 more Smart Citations

Improving Automatic Tuning of Hadoop and Spark by Analysing Container Performance Metrics

Zhou¹

Self Cite

View full text Add to dashboard Cite

This thesis attempts to provide further research on improving automatic tuning of Hadoop and Spark by analysing container performance metrics. Analytics frameworks, for example, Hadoop and Spark, are powerful tools in the study of big data. The parameter values in these analytics frameworks significantly affect the performance of applications. However, it is difficult to select the optimal framework parameter values for every application. Many research teams have proposed methods for the automatic tuning of Hadoop or Spark parameters from different perspectives. Most of the currently existing automatic tuning systems do not support working with different kinds of frameworks and Resource Managers. In addition, research on selecting the best timing to evoke the tuning method is lacking, which is another gap in the study of big data frameworks. Our research aims to fill these gaps. This research introduces novel container performance metrics and proves that these metrics are beneficial in the development of automatic tuning systems. Hadoop and Spark show different patterns in the static and dynamic values of container creation rate, container completion rate, container average response time and relative standard deviation of response-time(RSD). By applying five kinds of machine learning algorithms, container creation rate was found to be the most sensitive metric to identify and classify the workload type at an average accuracy of 83%. RSD can be used to detect workload transitions with an average accuracy of 74%. Our research results will decrease tuning overhead and promote the development of automatic tuning systems.

show abstract

Section: Thesis Contributionsmentioning

confidence: 99%

“…Several tests are running with initial parameter values and modified values. 2. Job performance and system resource usage logs are collected.…”

Section: Offline Auto-tuningmentioning

confidence: 99%

Section: Limitations Of Previous Researchmentioning

confidence: 99%

“…1. Choose an appropriate tuning timing by monitoring workload transitions 2. According to the current workload type, select a specific tuning strategy.…”

Section: Sectionmentioning

confidence: 99%

“…Client submits the application to the master node. 2. RM launches application master (AM) in the worker nodes.…”

Section: Introduction To Hadoop and Yarnmentioning

confidence: 99%

See 3 more Smart Citations