Straggler Root-Cause and Impact Analysis for Massive-scale Virtualized Cloud Datacenters

Abstract

Abstract-Increased complexity and scale of virtualized distributed systems has resulted in the manifestation of emergent phenomena substantially affecting overall system performance. This phenomena is known as "Long Tail", whereby a small proportion of task stragglers significantly impede job completion time. While work focuses on straggler detection and mitigation, there is limited work that empirically studies straggler root-cause and quantifies its impact upon system operation. Such analysis is critical to ascertain in-depth knowledge of straggler occurrence for focusing developmental and research efforts towards solving the Long Tail challenge. This paper provides an empirical analysis of straggler root-cause within virtualized Cloud datacenters; we analyze two large-scale production systems to quantify the frequency and impact stragglers impose, and propose a method for conducting root-cause analysis. Results demonstrate approximately 5% of task stragglers impact 50% of total jobs for batch processes, and 53% of stragglers occur due to high server resource utilization. We leverage these findings to propose a method for extreme straggler detection through a combination of offline execution patterns modeling and online analytic agents to monitor tasks at runtime. Experiments show the approach is capable of detecting stragglers less than 11% into their execution lifecycle with 95% accuracy for short duration jobs.

BigRoots: An Effective Approach for Root-Cause Analysis of Stragglers in Big Data System

“…We define straggler task as whose duration is 1.5× larger than median task duration [4,6,8] in the same stage. The idea of our root cause analysis is based on the following two observations: 1) If a feature is abnormal compared to tasks in other nodes, then this feature is highly possible the root cause of the straggler.…”

Optimal Server Selection for Straggler Mitigation

“…A slow server can delay the onset of next stage computation, and we call it a straggling server. One of the key challenges in cloud computing is the problem of straggling servers, which can significantly increase the job completion time [2]- [4]. Straggler mitigation is a particularly important problem, considering this the organizations such as VMWare and Amazon have spent substantial effort optimizing the operation of virtualization technologies for massive-scale systems [2].…”

“…One of the key challenges in cloud computing is the problem of straggling servers, which can significantly increase the job completion time [2]- [4]. Straggler mitigation is a particularly important problem, considering this the organizations such as VMWare and Amazon have spent substantial effort optimizing the operation of virtualization technologies for massive-scale systems [2]. This paper aims to find efficient scheduling mechanisms for straggler mitigation by analyzing how the replication of straggling tasks affects the mean service completion time and the mean server utilization cost of computing resources.…”

Influence of Tasks Duration Variability on Task-Based Runtime Schedulers

“…Re-execution is made easy by the use of distributed file systems based on replication such as HDFS [10]. The impact of stragglers and more generally of tasks that last longer than expected has been recently analyzed in [11] and our goal is to build the counterpart of this study in the context of HPC platforms. Indeed, this phenomena has been the object of very few studies in the context of dynamic schedulers on HPC platforms.…”