A Speculative Execution Framework for Big Data Processing Systems

Said, Samar A.; El-Sayed, M. Saad; Salem, Sameh A.; Habashy, Shahira M.

doi:10.1109/icit52682.2021.9491697

Cited by 2 publications

(2 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many parameters are considered by cluster utilization level coefficient such as CPU utilization, Memory utilization, Disk I/O utilization and Network bandwidth utilization as in Eqs. (10), (11), ( 12) and ( 13) [17].…”

Section: ) Dynamic Weighted Straggler Decision Criterionmentioning

confidence: 99%

“…But, MLlib Spark's is a scalable machine learning library. However, Spark is extremely recommended for data analytics, the appearance of Straggler tasks and performance deterioration in parallel systems may occur [17], [18]. In this context, a task is considered straggler when a machine has a significant delay to finish the execution of that task compared with other tasks at the similar stage [19], [20].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

An Optimized Straggler Mitigation Framework for Large-Scale Distributed Computing Systems

et al. 2022

Self Cite

View full text Add to dashboard Cite

Nowadays, Big Data becomes a research focus in industrial, banking, social network, and other fields. In addition, the explosive increase of data and information require efficient processing solutions. Therefore, Spark is considered as a promising candidate of Large-Scale Distributed Computing Systems for big data processing. One primary challenge is the straggler problem that occurred due to the presence of heterogeneity where a machine takes an extra-long time to finish execution of a task, which decreases the system throughput. To mitigate straggler tasks, Spark adopts speculative execution mechanism, in which the scheduler launches additional backup to avoid slow task processing and achieve acceleration. In this paper, a new Optimized Straggler Mitigation Framework is proposed. The proposed framework uses a dynamic criterion to determine the closest straggler tasks. This criterion is based on multiple coefficients to achieve a reliable straggler decision. Also, it integrates the historical data analysis and online adaptation for intelligent straggler judgment. This guarantees the effectiveness of speculative tasks by improving cluster performance. Experimental results on various benchmarks and applications show that the proposed framework achieves 23.5% to 30.7% execution time reductions, and 25.4 to 46.3% increase of the cluster throughputs compared with spark engine.

show abstract

Section: ) Dynamic Weighted Straggler Decision Criterionmentioning

confidence: 99%