Intra-query adaptivity for MapReduce query processing systems

Filho, Edson Ramiro Lucas; Almeida, Eduardo Cunha de; Traon, Yves Le

doi:10.1145/2628194.2628202

Cited by 4 publications

(1 citation statement)

References 5 publications

(3 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another search-based evolutionary computation approach was studied by Filho and colleagues in 2014 [37]. They proposed an adaptive tuning mechanism that enabled the setting of specific resources to each job within a query plan.…”

Section: Previous Workmentioning

confidence: 99%

Autonomic Performance Optimization of Big Data Workloads

Genkin¹

View full text Add to dashboard Cite

show abstract

Section: Previous Workmentioning

confidence: 99%

Autonomic Performance Optimization of Big Data Workloads

Genkin¹

View full text Add to dashboard Cite

show abstract

Real Time Autotuning for MapReduce on Hadoop/YARN

Pospelova¹

View full text Add to dashboard Cite

Humanity entered the Big Data era passing through the zettabite mark at an exponential rate in just a few years, which made Big Data processing issues a reality for a wide range of business and academic applications. The Apache Hadoop is the most popular Big Data processing framework with the current market share of $3 billion expected to increase to $50 billion by the end of the decade; however, Hadoop deployment expansion does not proceed at the predicted rate.One of the obstacles for further popularization of Hadoop is that performance is highly dependent on chosen settings. A nontrivial tuning process requires time and highly qualified human resources. Therefore, an automatic tuner is a desirable solution. Existent off-line tuning approaches require multiple repetitive executions of the job in order to find optimal tuning settings and, hence, are not applicable to use in most cases.The presented work introduces a novel real time autotuning approach. The resource management parameters are tuned between execution waves by a modular autotuner connected to Hadoop architecture through YARN. The developed autotuner effectively intercepts a resource request, modifies it according to a tuning algorithm and passes it to YARN Scheduler. Such an approach not only carries high practical and theoretical value, but also opens a new horizon in the Hadoop/YARN automatic tuning development.iii This thesis is dedicated to my husband, Pino Guerra, and to my children, Matteo and Isabella. Thank you for your love, patience, and numerous sacrifices that you have made throughout my academic career. This thesis and the pursuit of my goals would not have been possible without your support.iv

show abstract

Improving Real Time Tuning on YARN

Belmonte¹,

Pablo²

View full text Add to dashboard Cite

Big data is becoming a significant part of the operations of modern organizations. Performing analysis of large amounts of data requires computer clusters to run the calculations and analysis. YARN is an internal framework that is responsible for coordinating big data jobs for some popular distributed storage and processing frameworks like Hadoop and Spark. Running YARN with the correct configuration parameters is critical for the good performance of a cluster. KERMIT is an online tuner of YARN configuration parameters that aims to improve cluster performance. The first KER-MIT implementation proved the feasibility of the concept. In this study we modified the tuning algorithms inside the KERMIT components; by doing so, we achieved a reduction in the execution time as compared to industry benchmarks for a shallow tuning technique. We also verified that KERMIT can tune Spark; this suggests that Kermit could be used in other YARN-based frameworks.

show abstract

Intra-query adaptivity for MapReduce query processing systems

Cited by 4 publications

References 5 publications

Autonomic Performance Optimization of Big Data Workloads

Autonomic Performance Optimization of Big Data Workloads

Real Time Autotuning for MapReduce on Hadoop/YARN

Improving Real Time Tuning on YARN

Contact Info

Product

Resources

About