Straggler identification approach in large data processing frameworks using ensembled gradient boosting in smart-cities cloud services

Deshmukh, Shyam; Rao, K. Ram Mohan

doi:10.1007/s13198-021-01311-8

Cited by 1 publication

(1 citation statement)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This delay causes a degradation of the system throughput. Many studies introduced a speculative execution to overcome this problem, this leading to enhance the execution efficiency and the cluster performance [21] - [24]. In this context, late tasks can run on another nodes using speculative execution.…”

Section: Introductionmentioning

confidence: 99%

An Optimized Straggler Mitigation Framework for Large-Scale Distributed Computing Systems

et al. 2022

View full text Add to dashboard Cite

Nowadays, Big Data becomes a research focus in industrial, banking, social network, and other fields. In addition, the explosive increase of data and information require efficient processing solutions. Therefore, Spark is considered as a promising candidate of Large-Scale Distributed Computing Systems for big data processing. One primary challenge is the straggler problem that occurred due to the presence of heterogeneity where a machine takes an extra-long time to finish execution of a task, which decreases the system throughput. To mitigate straggler tasks, Spark adopts speculative execution mechanism, in which the scheduler launches additional backup to avoid slow task processing and achieve acceleration. In this paper, a new Optimized Straggler Mitigation Framework is proposed. The proposed framework uses a dynamic criterion to determine the closest straggler tasks. This criterion is based on multiple coefficients to achieve a reliable straggler decision. Also, it integrates the historical data analysis and online adaptation for intelligent straggler judgment. This guarantees the effectiveness of speculative tasks by improving cluster performance. Experimental results on various benchmarks and applications show that the proposed framework achieves 23.5% to 30.7% execution time reductions, and 25.4 to 46.3% increase of the cluster throughputs compared with spark engine.

show abstract

Section: Introductionmentioning

confidence: 99%