Apache Hadoop Yarn Parameter configuration Challenges and Optimization

Mathiya, Bhavin J.; Desai, Vinodkumar L.

doi:10.1109/icsns.2015.7292373

“…Uma compilação de diversos estudos sobre frameworks que utilizam parâmetros de con guração para melhorar o desempenho do Hadoop foi apresentado no trabalho de Mathiya and Desai (2015). Os autores identi caram vários parâmetros e apontaram aqueles que são mais relevantes, dividindo-os em três grupos, chamados CPU, Memória e I/O.…”

Section: Trabalhos Relacionadosunclassified

“…O framework Hadoop possui mais de 200 parâmetros de con guração e atribuir valores de maneira otimizada para eles não é uma tarefa trivial, devido à grande quantidade de combinações que podem ser formadas (Chen et al;. Customizar os parâmetros de con guração é uma tarefa que requer bons conhecimentos de cada um deles, pois a mudança do valor de um parâmetro pode impactar em outros, uma vez que eles são interconectados entre si (Mathiya and Desai;2015). Diversos estudos da literatura sugerem valores para esses parâmetros, contudo existe uma lacuna para recomendações de melhores práticas para con guração de frameworks distribuídos (Li et al; (Garvit et al; (Guo and Fox; (Krishna et al;.…”

Section: Introductionunclassified

Projeto e análise de desempenho de um algoritmo iterativo para grandes grafos em um ambiente distribuído

Nascimento¹,

Capanema²,

Pereira³

2019

RBCA

0

View full text Add to dashboard Cite

Atualmente grandes volumes de dados são gerados e coletados por meio de sensores, dispositivos e redes sociais. A capacidade de lidar com grandes massas de dados tornou-se um importante fator para o sucesso de muitas organizações, exigindo, cada vez mais, a utilização de processamento paralelo e distribuído. Para auxiliar os desenvolvedores a projetar programas distribuídos, existem várias ferramentas (frameworks), como Apache Hadoop e Spark. Esses frameworks fornecem diversos parâmetros de configuração (por exemplo, o Hadoop tem mais de 200) e atribuir valores otimizados a todos eles não é uma tarefa simples. Este trabalho investiga a influência desses parâmetros no desempenho do Apache Hadoop, utilizando o algoritmo HEDA, um algoritmo iterativo que calcula métricas de centralidade em grandes grafos. A execução do HEDA em uma rede complexa é extremamente importante, pois existem várias medidas de centralidade que determinam a importância de um vértice dentro do grafo. Observou-se que, em alguns casos, a melhoria no tempo de execução atingiu aproximadamente 80% aplicando os valores propostos por este trabalho aos parâmetros de configuração do Hadoop. Além disso, foi possível aumentar em cinco vezes o uso dos processadores e melhorar consideravelmente a escalabilidade. O trabalho também apresenta os métodos aplicados para preparar, executar e analisar os experimentos, o que poderá auxiliar em novos estudos.

show abstract

“…[26] [14][15] [62][120] [109][110] [111][108]. The statistical significance of the results validate the differences in the performance of the two experiments.…”

mentioning

confidence: 85%

“…To optimize Hadoop job performance and resource utilization, we should carefully select from an extensive list of 200 parameters [18]. Other attributes such as job input, dataset, cluster size and infrastructure can also be used to model its performance and resource utilization [18] [26] [14] [15] [62] [120]. The inter-related impact of these attributes can be hard to predict manually.…”

Section: Research Goal and Objectivesmentioning

confidence: 99%

Resource and Performance Modelling of Hadoop Clusters Using Machine Learning

Tariq¹

0

View full text Add to dashboard Cite

<p>There is a huge and rapidly increasing amount of data being generated by social media, mobile applications and sensing devices. Big data is the term usually used to describe such data and is described in terms of the 3Vs - volume, variety and velocity. In order to process and mine such a massive amount of data, several approaches and platforms have been developed such as Hadoop. Hadoop is a popular open source distributed and parallel computing framework. It has a large number of configurable parameters which can be set before the execution of jobs to optimize the resource utilization and execution time of the clusters. These parameters have a significant impact on system resources and execution time. Optimizing the performance of a Hadoop cluster by tuning such a large number of parameters is a tedious task. Most current big data modeling approaches do not include the complex interaction between configuration parameters and the cluster environment changes such as use of different datasets or types of query. This makes it difficult to predict for example the execution time of a job or resource utilization of a cluster. Other attributes include configuration parameters, the structure of query, the dataset, number of nodes and the infrastructure used. Our first main objective was to design reliable experiments to understand the relationship between attributes. Before designing and implementing the actual experiment we applied Hazard and Operability (HAZOP) analysis to identify operational hazards. These hazards can affect normal working of cluster and execution of Hadoop jobs. This brainstorming activity improved the design and implementation of our experiments by improving the internal validity of the experiments. It also helped us to identify the considerations that must be taken into account for reliable results. After implementing our design, we characterized the relationship between different Hadoop configuration parameters, network and system performance measures. Our second main objective was to investigate the use of machine learning to model and predict the resource utilization and execution time of Hadoop jobs. Resource utilization and execution time of Hadoop jobs are affected by different attributes such as configuration parameters and structure of query. In order to estimate or predict either qualitatively or quantitatively the level of resource utilization and execution time, it is important to understand the impact of different combinations of these Hadoop job attributes. You could conduct experiments with many different combinations of parameters to uncover this but it is very difficult to run such a large number of jobs with different combinations of Hadoop job attributes and then interpret the data manually. It is very difficult to extract patterns from the data and give a model that can generalize for an unseen scenario. In order to automate the process of data extraction and modeling the complex behavior of different attributes of Hadoop job machine learning was used. Our decision tree based approach enabled us to systematically discover significant patterns in data. Our results showed that the decision tree models constructed for different resources and execution time were informative and robust. They were able to generalize over a wide range of minor and major environmental changes such as change in dataset, cluster size and infrastructure such as Amazon EC2. Moreover, the use of different correlation and regression techniques, such as M5P, Pearson's correlation and k-means clustering, confirmed our findings and provided further insight into the relationship of different attributes and with each other. M5P is a classification and regression technique that predicted the functional relationships among different job attributes. The use of k-means clustering allowed us to see the experimental runs that shows similar resource utilization and execution time. Statistical significance tests, were used to validate the significance of changes in results of different experimental runs, also showed the effectiveness of our resource and performance modelling and prediction method.</p>

show abstract

“…Mathiya and Desai [120] demonstrated that the performance of Apache Hadoop Yet Another Resource Negotiator (YARN) depends upon a large number of parameters. They classified the Hadoop YARN parameter configuration settings into different categories.…”

Section: Parametersmentioning

confidence: 99%

Resource and Performance Modelling of Hadoop Clusters Using Machine Learning

Tariq¹

0

View full text Add to dashboard Cite

<p>There is a huge and rapidly increasing amount of data being generated by social media, mobile applications and sensing devices. Big data is the term usually used to describe such data and is described in terms of the 3Vs - volume, variety and velocity. In order to process and mine such a massive amount of data, several approaches and platforms have been developed such as Hadoop. Hadoop is a popular open source distributed and parallel computing framework. It has a large number of configurable parameters which can be set before the execution of jobs to optimize the resource utilization and execution time of the clusters. These parameters have a significant impact on system resources and execution time. Optimizing the performance of a Hadoop cluster by tuning such a large number of parameters is a tedious task. Most current big data modeling approaches do not include the complex interaction between configuration parameters and the cluster environment changes such as use of different datasets or types of query. This makes it difficult to predict for example the execution time of a job or resource utilization of a cluster. Other attributes include configuration parameters, the structure of query, the dataset, number of nodes and the infrastructure used. Our first main objective was to design reliable experiments to understand the relationship between attributes. Before designing and implementing the actual experiment we applied Hazard and Operability (HAZOP) analysis to identify operational hazards. These hazards can affect normal working of cluster and execution of Hadoop jobs. This brainstorming activity improved the design and implementation of our experiments by improving the internal validity of the experiments. It also helped us to identify the considerations that must be taken into account for reliable results. After implementing our design, we characterized the relationship between different Hadoop configuration parameters, network and system performance measures. Our second main objective was to investigate the use of machine learning to model and predict the resource utilization and execution time of Hadoop jobs. Resource utilization and execution time of Hadoop jobs are affected by different attributes such as configuration parameters and structure of query. In order to estimate or predict either qualitatively or quantitatively the level of resource utilization and execution time, it is important to understand the impact of different combinations of these Hadoop job attributes. You could conduct experiments with many different combinations of parameters to uncover this but it is very difficult to run such a large number of jobs with different combinations of Hadoop job attributes and then interpret the data manually. It is very difficult to extract patterns from the data and give a model that can generalize for an unseen scenario. In order to automate the process of data extraction and modeling the complex behavior of different attributes of Hadoop job machine learning was used. Our decision tree based approach enabled us to systematically discover significant patterns in data. Our results showed that the decision tree models constructed for different resources and execution time were informative and robust. They were able to generalize over a wide range of minor and major environmental changes such as change in dataset, cluster size and infrastructure such as Amazon EC2. Moreover, the use of different correlation and regression techniques, such as M5P, Pearson's correlation and k-means clustering, confirmed our findings and provided further insight into the relationship of different attributes and with each other. M5P is a classification and regression technique that predicted the functional relationships among different job attributes. The use of k-means clustering allowed us to see the experimental runs that shows similar resource utilization and execution time. Statistical significance tests, were used to validate the significance of changes in results of different experimental runs, also showed the effectiveness of our resource and performance modelling and prediction method.</p>

show abstract

Apache Hadoop Yarn Parameter configuration Challenges and Optimization

Cited by 22 publications

References 14 publications

Projeto e análise de desempenho de um algoritmo iterativo para grandes grafos em um ambiente distribuído

Projeto e análise de desempenho de um algoritmo iterativo para grandes grafos em um ambiente distribuído

Resource and Performance Modelling of Hadoop Clusters Using Machine Learning

Resource and Performance Modelling of Hadoop Clusters Using Machine Learning

Contact Info

Product

Resources

About