Computing Resource Prediction for MapReduce Applications Using Decision Tree

Piao, Jing Tai; Yan, Jun

doi:10.1007/978-3-642-29253-8_51

Cited by 10 publications

(8 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…PQR2, an approach to accurate performance evaluation of distributed application in a cloud, was presented . A model for predicting resource consumption of MapReduce processes was set up based on a classification and regression tree .…”

Section: Related Workmentioning

confidence: 99%

“…While the optimization of job scheduling in MapReduce has been widely conducted in recent activities , current Hadoop systems still suffer from poor load‐scheduling strategies because of their lack of consideration on the usage of cloud storage, which would bring heavy loads on certain data nodes and therefore cause a long delay on total execution. Although theoretically infinite computing resources can be provided in a cloud system, unreasonable increment of mappers/reducers cannot achieve processing efficiency and even waste more storage to complete.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A speculative approach to spatial‐temporal efficiency with multi‐objective optimization in a heterogeneous cloud environment

Liu

Cai

Shen

et al. 2016

Security Comm Networks

196

View full text Add to dashboard Cite

A heterogeneous cloud system, for example, a Hadoop 2.6.0 platform, provides distributed but cohesive services with rich features on large‐scale management, reliability, and error tolerance. As big data processing is concerned, newly built cloud clusters meet the challenges of performance optimization focusing on faster task execution and more efficient usage of computing resources. Presently proposed approaches concentrate on temporal improvement, that is, shortening MapReduce time, but seldom focus on storage occupation; however, unbalanced cloud storage strategies could exhaust those nodes with heavy MapReduce cycles and further challenge the security and stability of the entire cluster. In this paper, an adaptive method is presented aiming at spatial–temporal efficiency in a heterogeneous cloud environment. A prediction model based on an optimized Kernel‐based Extreme Learning Machine algorithm is proposed for faster forecast of job execution duration and space occupation, which consequently facilitates the process of task scheduling through a multi‐objective algorithm called time and space optimized NSGA‐II (TS‐NSGA‐II). Experiment results have shown that compared with the original load‐balancing scheme, our approach can save approximate 47–55 s averagely on each task execution. Simultaneously, 1.254‰ of differences on hard disk occupation were made among all scheduled reducers, which achieves 26.6% improvement over the original scheme. Copyright © 2016 John Wiley & Sons, Ltd.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A speculative approach to spatial‐temporal efficiency with multi‐objective optimization in a heterogeneous cloud environment

Liu

Cai

Shen

et al. 2016

Security Comm Networks

196

View full text Add to dashboard Cite

show abstract

“…PQR2, an approach to accurate performance evaluation of distributed application in a cloud was presented [14]. Jing et al presented a model that can predict resource consumption of MapReduce processes based on a classification and regression tree [15].…”

Section: Related Workmentioning

confidence: 99%

“…Several projects have been launched to relieve the difficulty in writing complex data analysis or data mining programs, e.g., Pig [2] and Hive [3] built upon the MapReduce engines in the Hadoop environment. Optimization schemes have also been proposed [4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19].…”

Section: Introductionmentioning

confidence: 99%

A Smart Strategy for Speculative Execution Based on Hardware Resource in a Heterogeneous Distributed Environment

Liu¹,

Cai²,

Fu³

et al. 2016

IJGDC

View full text Add to dashboard Cite

“…They estimated the resource usage for an application by an approach called PQR2. Jing et al [10] presented a model that can predict the computing resource consumption of MapReduce applications based on a Classified and Regression Tree.…”

Section: Related Workmentioning

confidence: 99%