A New Framework for Evaluating Straggler Detection Mechanisms in MapReduce

Abstract

Big Data systems (e.g., Google MapReduce, Apache Hadoop, Apache Spark) rely increasingly on speculative execution to mask slow tasks, also known as stragglers, because a job's execution time is dominated by the slowest task instance. Big Data systems typically identify stragglers and speculatively run copies of those tasks with the expectation that a copy may complete faster to shorten job execution times. There is a rich body of recent results on straggler mitigation in MapReduce. However, the majority of these do not consider the problem of accurately detecting stragglers. Instead, they adopt a particular straggler detection approach and then study its effectiveness in terms of performance, e.g., reduction in job completion time, or efficiency, e.g., high resource utilization. In this paper, we consider a complete framework for straggler detection and mitigation. We start with a set of metrics that can be used to characterize and detect stragglers including Precision, Recall, Detection Latency, Undetected Time and Fake Positive. We then develop an architectural model by which these metrics can be linked to measures of performance including execution time and system energy overheads. We further conduct a series of experiments to demonstrate which metrics and approaches are more effective in detecting stragglers and are also predictive of effectiveness in terms of performance and energy efficiencies. For example, our results indicate that the default Hadoop straggler detector could be made more effective. In certain case, Precision is low and only 55% of those detected are actual stragglers and the Recall, i.e., percent of actual detected stragglers, is also relatively low at 56%. For the same case, the hierarchical approach (i.e., a green-driven detector based on the default one) achieves a Precision of 99% and a Recall of 29%. This increase in Precision can be translated to achieve lower execution time and energy consumption, and thus higher performance and energy efficiency; compared to the default Hadoop mechanism, the energy consumption is reduced by almost 31%. These results demonstrate how our framework can offer useful insights and be applied in practical settings to characterize and design new straggler detection mechanisms for MapReduce systems. This work is supported by the ANR KerStream project (ANR-16-CE25-0014-01) and the Stack/Apollo connect talent project. The experiments presented in this paper were carried out using the Grid'5000/ALADDIN-G5K experimental testbed, an initiative from the French Ministry of Research through the ACI GRID incentive action, INRIA, CNRS and RENATER and other contributing partners (see http://www.grid5000.fr/ for details).

FLeet

“…In order to accelerate learning, AdaSGD relies on a system parameter: the expected percentage of nonstragglers (denoted by s%). We highlight that this value is not a hyperparameter that needs tuning for each ML application but a system parameter that solely depends on the computing and networking characteristics of the workers, while it can be adapted dynamically [64,65]. We define the staleness-aware dampening factor Λ(τ ) = e −βτ , with β chosen s.t.…”

Prism

“…The framework was evaluated within an 8-node Hadoop cluster, whereby they injected stragglers to alleviate job data skew. Tien-Dat et al [16] proposed a framework for straggler detection and mitigation to enhance job execution time and system energyefficiency. Using the Grid'5000 testbed consisting of 21-nodes, authors artificially injected stragglers into job application execution and evaluated the framework with straggler mitigation techniques.…”

“…Conducting such experiments allows for empirical study of realistic system operation in order to propose new approaches without interfering with production system behavior, as well as underpin parameterization of Cloud simulation frameworks [9]. This is particularly important as current simulators are unable to realistically represent straggler manifestation due to their decoupling of occurrence probability and the underlying cluster operational conditions [14][15][16].…”

“…This also imposes additional limitations on researchers, reducing the number of approaches that can be feasibly compared for evaluation. These issues have resulted in a large body of research relying on small-scale experiments [15] or simulated environments [14] [16]. Although such approaches are appropriate for designing new straggler mitigation techniques, simulated environments make assumptions about underlying causes of straggler manifestation and require expert knowledge to achieve representative behaviors.…”

Influence of Tasks Duration Variability on Task-Based Runtime Schedulers

“…Related Works: The problem of tasks duration variability has received significant attention in the context of distributed computing, first in Grid platforms [19] and later in Cloud computing platforms in the context of MapReduce environment [20]. Some of the work in this area consists one one hand in analyzing the variability of task execution times and trying to explain it [21,22]. On the other hand, several strategies have been designed to mitigate the effect of variability.…”