MrHeter: improving MapReduce performance in heterogeneous environments

Xiao, Zhang; Wu, Yicheng; Zhao, Chen

doi:10.1007/s10586-016-0625-2

Cited by 39 publications

(16 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As data are transferred in the copy and shuffle stages, they have the most significant impact at execution time. There is an assigned weight for each stage, which is the ratio of the stage's execution time to the total execution time [28,29,32,33,34,35,36]. It is also possible to calculate errors by comparing the estimated weights by a real one [15].…”

Section: Introductionmentioning

confidence: 99%

RETRACTED ARTICLE: Detecting straggler MapReduce tasks in big data processing infrastructure by neural network

et al. 2020

View full text Add to dashboard Cite

show abstract

Section: Introductionmentioning

confidence: 99%

RETRACTED ARTICLE: Detecting straggler MapReduce tasks in big data processing infrastructure by neural network

et al. 2020

View full text Add to dashboard Cite

show abstract

“…On the other hand, performance parameters and utilization of the available nodes can affect the overall task processing. In [28] some of the issues related to MapReduce performance in distributed environments are addressed in heterogeneous clusters, with authors focusing on the unreasonable allocation of tasks to the nodes with different computational capabilities to prove that such optimization brings significant benefits and greatly improves the efficiency of MapReduce-based algorithms. Authors in [29] solve the problem of task assignment and resource allocation in distributed systems using a genetic algorithm.…”

Section: Optimization Of Task Assignment In Distributed Environmentsmentioning

confidence: 99%

Improvement in the Efficiency of a Distributed Multi-Label Text Classification Algorithm Using Infrastructure and Task-Related Data

Sarnovský

Olejnik

2019

Informatics

View full text Add to dashboard Cite

Distributed computing technologies allow a wide variety of tasks that use large amounts of data to be solved. Various paradigms and technologies are already widely used, but many of them are lacking when it comes to the optimization of resource usage. The aim of this paper is to present the optimization methods used to increase the efficiency of distributed implementations of a text-mining model utilizing information about the text-mining task extracted from the data and information about the current state of the distributed environment obtained from a computational node, and to improve the distribution of the task on the distributed infrastructure. Two optimization solutions are developed and implemented, both based on the prediction of the expected task duration on the existing infrastructure. The solutions are experimentally evaluated in a scenario where a distributed tree-based multi-label classifier is built based on two standard text data collections.

show abstract

“…Default Hadoop places data uniformly inside the cluster assuming that all nodes are homogeneous [6].…”

Section: Introductionmentioning

confidence: 99%

A counter based approach for reducer placement with augmented Hadoop rack awareness

Hussain¹,

Reddy²,

Roy³

2021

Turk J Elec Eng & Comp Sci

View full text Add to dashboard Cite

As the data-driven paradigm for intelligent systems design is gaining prominence, performance requirements have become very stringent, leading to numerous fine-tuned versions of Hadoop and its MapReduce programming model. However, very few researchers have investigated the effect of intelligent reducer placement on Hadoop's performance. This paper delves into this much ignored reducer placement phase for improving Hadoop's performance and proposes to spawn reduce phase of Hadoop tasks in an asynchronous fashion across nodes in a Hadoop cluster. The main contributions of this paper are: (i) to track when map phase of tasks are completed, (ii) to count the number of maps completed, and finally (iii) assign reducers to Hadoop nodes based on map counts such that run-time data copying is minimized. To this end, this paper presents a novel counter based reducer placement (CBRP) algorithm based on the counter values maintained by JobTracker at the rack and node levels. Experiments conducted demonstrate the merit of the proposed reducer placement with average improvements ranging between 5% and 17% experienced across different benchmarks with both late shuffle and early shuffle.

show abstract

MrHeter: improving MapReduce performance in heterogeneous environments

Cited by 39 publications

References 17 publications

RETRACTED ARTICLE: Detecting straggler MapReduce tasks in big data processing infrastructure by neural network

RETRACTED ARTICLE: Detecting straggler MapReduce tasks in big data processing infrastructure by neural network

Improvement in the Efficiency of a Distributed Multi-Label Text Classification Algorithm Using Infrastructure and Task-Related Data

A counter based approach for reducer placement with augmented Hadoop rack awareness

Contact Info

Product

Resources

About