Scalable Performance Tuning of Hadoop MapReduce: A Noisy Gradient Approach

Kumar, Sandeep; Padakandla, Sindhu; Chandrashekar, L; Parihar, Priyank; Gopinath, K.; Bhatnagar, Shalabh

doi:10.1109/cloud.2017.55

Cited by 12 publications

(5 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The goal of the SPSA algorithm in this article is to maximize the objective function f (θ) by updating the configuration parameters θ of the Elasticsearch over time. In order to optimize the configuration we resort to a modified version of the Simultaneous Perturbation Stochastic Approximation algorithm (Kumar et al, 2017 ). The implemented algorithm in this project is presented in Algorithm 1 .…”

Section: Methodsmentioning

confidence: 99%

Tuning of Elasticsearch Configuration: Parameter Optimization Through Simultaneous Perturbation Stochastic Approximation

2022

View full text Add to dashboard Cite

Elasticsearch is currently the most popular search engine for full-text database management systems. By default, its configuration does not change while it receives data. However, when Elasticsearch stores a large amount of data over time, the default configuration becomes an obstacle to improving performance. In addition, the servers that host Elasticsearch may have limited resources, such as internal memory and CPU. A general solution to these problems is to dynamically tune the configuration parameters of Elasticsearch in order to improve its performance. The sheer number of parameters involved in this configuration makes it a complex task. In this work, we apply the Simultaneous Perturbation Stochastic Approximation method for optimizing Elasticsearch with multiple unknown parameters. Using this algorithm, our implementation optimizes the Elasticsearch configuration parameters by observing the performance and automatically changing the configuration to improve performance. The proposed solution makes it possible to change the configuration parameters of Elasticsearch automatically without having to restart the currently running instance of Elasticsearch. The results show a higher than 40% improvement in the combined data insertion capacity and the system's response time.

show abstract

Section: Methodsmentioning

confidence: 99%

Tuning of Elasticsearch Configuration: Parameter Optimization Through Simultaneous Perturbation Stochastic Approximation

2022

View full text Add to dashboard Cite

show abstract

“…A recent related study of the search-based method is BestConfig [15], which uses the divide-and-diverge sampling method and the recursive bound-and-search algorithm to automatically tune configurations with limited resources for general systems. Kumar et al [16] propose a noise-gradient algorithm, called simultaneous perturbation stochastic approximation (SPSA), to optimize Hadoop's performance. Moreover, several studies explore the search-based method, such as ACTS [17] and MRONLINE [18].…”

Section: Related Workmentioning

confidence: 99%

ATCS: Auto-Tuning Configurations of Big Data Frameworks Based on Generative Adversarial Nets

Liu

Shi

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Big data processing frameworks (e.g., Spark, Storm) have been extensively used for massive data processing in the industry. To improve the performance and robustness of these frameworks, developers provide users with highly-configurable parameters. Due to the high-dimensional parameter space and complicated interactions of parameters, manual tuning of parameters is time-consuming and ineffective. Building performance-predicting models for big data frameworks is challenging for several reasons: (1) the significant time required to collect training data and (2) the poor accuracy of the prediction model when training data are limited. To meet this challenge, we proposes an auto-tuning configuration parameters system (ATCS), a new auto-tuning approach based on Generative Adversarial Nets (GAN). ATCS can build a performance prediction model with less training data and without sacrificing model accuracy. Moreover, an optimized Genetic Algorithm (GA) is used in ATCS to explore the parameter space for optimum solutions. To prove the effectiveness of ATCS, we select five frequently-used workloads in Spark, each of which runs on five different sized data sets. The results demonstrate that ATCS improves the performance of five frequently-used Spark workloads compared to the default configurations. We achieved a performance increase of 3.5× on average, with a maximum of 6.9×. To obtain similar model accuracy, experiment results also demonstrate that the quantity of ATCS training data is only 6% of Deep Neural Network (DNN) data, 13% of Support Vector Machine (SVM) data, 18% of Decision Tree (DT) data. Moreover, compared to other machine learning models, the average performance increase of ATCS is 1.7× that of DNN, 1.6× that of SVM, 1.7× that of DT on the five typical Spark programs. INDEX TERMS Big data, generative adversarial nets, spark, genetic algorithm, automatic tune parameters.

show abstract

“…Os autores Kumar et al (2016) usaram um algoritmo Noisy Gradient para otimizar valores para os parâmetros de con guração e alcançaram melhorias no tempo de execução próximas a 66% quando comparados com a con guração padrão do Hadoop.…”

Section: Trabalhos Relacionadosunclassified

Projeto e análise de desempenho de um algoritmo iterativo para grandes grafos em um ambiente distribuído

Nascimento¹,

Capanema²,

Pereira³

2019

RBCA

View full text Add to dashboard Cite

Atualmente grandes volumes de dados são gerados e coletados por meio de sensores, dispositivos e redes sociais. A capacidade de lidar com grandes massas de dados tornou-se um importante fator para o sucesso de muitas organizações, exigindo, cada vez mais, a utilização de processamento paralelo e distribuído. Para auxiliar os desenvolvedores a projetar programas distribuídos, existem várias ferramentas (frameworks), como Apache Hadoop e Spark. Esses frameworks fornecem diversos parâmetros de configuração (por exemplo, o Hadoop tem mais de 200) e atribuir valores otimizados a todos eles não é uma tarefa simples. Este trabalho investiga a influência desses parâmetros no desempenho do Apache Hadoop, utilizando o algoritmo HEDA, um algoritmo iterativo que calcula métricas de centralidade em grandes grafos. A execução do HEDA em uma rede complexa é extremamente importante, pois existem várias medidas de centralidade que determinam a importância de um vértice dentro do grafo. Observou-se que, em alguns casos, a melhoria no tempo de execução atingiu aproximadamente 80% aplicando os valores propostos por este trabalho aos parâmetros de configuração do Hadoop. Além disso, foi possível aumentar em cinco vezes o uso dos processadores e melhorar consideravelmente a escalabilidade. O trabalho também apresenta os métodos aplicados para preparar, executar e analisar os experimentos, o que poderá auxiliar em novos estudos.

show abstract

Scalable Performance Tuning of Hadoop MapReduce: A Noisy Gradient Approach

Cited by 12 publications

References 17 publications

Tuning of Elasticsearch Configuration: Parameter Optimization Through Simultaneous Perturbation Stochastic Approximation

Tuning of Elasticsearch Configuration: Parameter Optimization Through Simultaneous Perturbation Stochastic Approximation

ATCS: Auto-Tuning Configurations of Big Data Frameworks Based on Generative Adversarial Nets

Projeto e análise de desempenho de um algoritmo iterativo para grandes grafos em um ambiente distribuído

Contact Info

Product

Resources

About