A Smart Strategy for Speculative Execution Based on Hardware Resource in a Heterogeneous Distributed Environment

Liu, Qi; Cai, Weidong; Fu, Zhangjie; Shen, Jian; Linge, Nigel

doi:10.14257/ijgdc.2016.9.2.18

Cited by 6 publications

(5 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On top of that, there are some optimization methods put forward. A execution was proposed based on sort nodes out according to the hardware performance of the nodes [22]. Wang et al proposed a PSE optimization strategy that can ignore the differences between different processors to improve the efficiency of speculative execution.…”

Section: Fault Tolerance In Hadoopmentioning

confidence: 99%

Near-Data Prediction Based Speculative Optimization in a Distribution Environment

Sun

Jin

et al. 2020

Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

Self Cite

View full text Add to dashboard Cite

ApacheHadoop is an open source software framework that supports data-intensive distributed applications and is distributed under the Apache 2.0 licensing agreement, where consumers will no longer deal with complex configuration of software and hardware but only pay for cloud services on demand. So how to make the performance of the cloud platform become more important in a consumer-centric environment. There exists imbalance between in some distribution of slow tasks, which results in straggling tasks will have a great influence on the Hadoop framework. By monitoring those tasks in real-time progress and copying the potential Stragglers to a different node, the speculative execution (SE) realizes to improve the probability of finishing those backup tasks before the original ones. The Speculative execution (SE) applies this principle and thus proposed a solution to handle the Straggling tasks. At present, the performance of the Hadoop system is unsatisfying because of the erroneous judgement and inappropriate selection for the backup nodes in the current SE policy. This paper proposes an SE optimized strategy which can be used in prediction of near data. In this strategy, the first step is gathering the real-time task execution information and the remaining runtime required for the task is predicted by a local prediction method. Then it chooses a proper backup node according to the near data and actual demand in the second step. On the other side, this model also includes a cost-effective model in order to make the performance of SE to the peak. The results show that using this strategy in Hadoop effectively improves the accuracy of alternative tasks and effects better in heterogeneous Hadoop environments in various situations, which is beneficial to consumers and cloud platform.

show abstract

Section: Fault Tolerance In Hadoopmentioning

confidence: 99%

Near-Data Prediction Based Speculative Optimization in a Distribution Environment

Sun

Jin

et al. 2020

Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

Self Cite

View full text Add to dashboard Cite

show abstract

“…As the most critical part in MR, scheduler’s efficiency decides whether data can be processed efficiently and tremendous progress has been made now [ 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 ]. Apart from the scheduler, speculative execution strategies also gained wide attention with an increasing number of researchers concentrating on optimizing the performance of speculative execution [ 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 ].…”

Section: Related Workmentioning

confidence: 99%

“…Recently, speculative execution strategies (SE) have been proposed [ 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 ]. Due to the unreasonable scheduling algorithms, SE strategy is used for solving the struggles and usually seen as a fault-tolerant mechanism.…”

Section: Related Workmentioning

confidence: 99%

“…An extended MCP was proposed while a load curve was added [ 28 ]. A smart strategy was proposed based on hardware performance and data volume of each phase [ 29 ]. In [ 30 ], the differences of work nodes were investigated to estimate backup tasks precisely.…”

Section: Related Workmentioning

confidence: 99%

“…Previous work has been tremendously conducted on optimizing the Scheduler [ 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 ]. Apart from the scheduler, speculative execution strategies have also gained wide attention [ 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 ], since incorrect estimating the running duration of a run-time task may cause its inappropriate allocation. For periodically executed jobs, an optimized speculative execution strategy can effectively improve the performance of entire MR processing.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Estimation Accuracy on Execution Time of Run-Time Tasks in a Heterogeneous Distributed Environment

Liu

Cai

Jin

et al. 2016

Sensors

Self Cite

View full text Add to dashboard Cite

Distributed Computing has achieved tremendous development since cloud computing was proposed in 2006, and played a vital role promoting rapid growth of data collecting and analysis models, e.g., Internet of things, Cyber-Physical Systems, Big Data Analytics, etc. Hadoop has become a data convergence platform for sensor networks. As one of the core components, MapReduce facilitates allocating, processing and mining of collected large-scale data, where speculative execution strategies help solve straggler problems. However, there is still no efficient solution for accurate estimation on execution time of run-time tasks, which can affect task allocation and distribution in MapReduce. In this paper, task execution data have been collected and employed for the estimation. A two-phase regression (TPR) method is proposed to predict the finishing time of each task accurately. Detailed data of each task have drawn interests with detailed analysis report being made. According to the results, the prediction accuracy of concurrent tasks’ execution time can be improved, in particular for some regular jobs.

show abstract

A Hybrid Resource Scheduling Strategy in Speculative Execution Based on Non-cooperative Game Theory

Dannah

Liu

Jin

2018

Cloud Computing and Security

View full text Add to dashboard Cite

A Smart Strategy for Speculative Execution Based on Hardware Resource in a Heterogeneous Distributed Environment

Cited by 6 publications

References 22 publications

Near-Data Prediction Based Speculative Optimization in a Distribution Environment

Near-Data Prediction Based Speculative Optimization in a Distribution Environment

Estimation Accuracy on Execution Time of Run-Time Tasks in a Heterogeneous Distributed Environment

A Hybrid Resource Scheduling Strategy in Speculative Execution Based on Non-cooperative Game Theory

Contact Info

Product

Resources

About