Analytical Performance Models for MapReduce Workloads

Vianna, Emanuel; Comarela, Giovanni; Pontes, Tatiana Barcelos; Almeida, Jussara M.; Almeida, Virgílio A. F.; Wilkinson, Kevin; Kuno, Harumi; Dayal, Umeshwar

doi:10.1007/s10766-012-0227-4

Cited by 53 publications

(49 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using the model from [37] the work in [38] provides approximations for the number of jobs in a tandem system consisting of a map queue and a reduce queue in the heavy traffic regime. The work in [41] derives approximations of the mean response time in MapReduce systems using a mean value analysis technique and a closed FJ queueing system model from [39].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Stochastic bounds in Fork–Join queueing systems under full and partial mapping

2016

View full text Add to dashboard Cite

In a Fork-Join (FJ) queueing system an upstream fork station splits incoming jobs into N tasks to be further processed by N parallel servers, each with its own queue; the response time of one job is determined, at a downstream join station, by the maximum of the corresponding tasks' response times. This queueing system is useful to the modelling of multi-service systems subject to synchronization constraints, such as MapReduce clusters or multipath routing. Despite their apparent simplicity, FJ systems are hard to analyze.This paper provides the first computable stochastic bounds on the waiting and response time distributions in FJ systems under full (bijective) and partial (injective) mapping of tasks to servers. We consider four practical scenarios by combining 1a) renewal and 1b) non-renewal arrivals, and 2a) non-blocking and 2b) blocking servers. In the case of non-blocking servers we prove that delays scale as O(log N ), a law which is known for first moments under renewal input only. In the case of blocking servers, we prove that the same factor of log N dictates the stability region of the system. Simulation results indicate that our bounds are tight, especially at high utilizations, in all four scenarios. A remarkable insight gained from our results is that, at moderate to high utilizations, multipath routing "makes sense" from a queueing perspective for two paths only, i.e., response times drop the most when N = 2; the technical explanation is that the resequencing (delay) price starts to quickly dominate the tempting gain due to multipath transmissions.

show abstract

Section: Related Workmentioning

confidence: 99%

“…h (2) prior to N ; moreover, θ is the implicit solution from (41). Figure 10 illustratesR N as a function of N for several utilization levels ρ for both renewal (a) and non-renewal (b) input; recall that the utilization on each path is ρ N .…”

Section: Non-renewal Arrivalsmentioning

confidence: 99%

Stochastic bounds in Fork–Join queueing systems under full and partial mapping

2016

View full text Add to dashboard Cite

show abstract

“…Some other approaches, e.g., [24], are based on an approximate mean value analysis technique and use an iterative hierarchical approach. Along the same lines, [34] combines a precedence graph and a QN to capture the intra-job synchronization constraints, thus being able to estimate the synchronization delays introduced by the communication among mappers and reducers. Unfortunately, even if the approach is rather accurate (around 15% accuracy on real systems), the authors assume that CPU slots are statically assigned to mappers and reducers, hence the proposed method cannot be adopted to estimate performance under the Hadoop 2.x dynamic resource assignment policy.…”

Section: Modeling Hadoop 2x Applications Performancementioning

confidence: 99%

“…MapReduce has been adopted in multiple application domains, e.g., machine learning, graph processing, and data mining [34], and its open source implementation, Hadoop 2.x, recently introduced a wide set of performance enhancements (e.g., SSD support, caching, and I/O barriers mitigation). IDC estimates that Hadoop touched half of the world data last year [20] supporting both traditional batch and interactive data analysis applications [32].…”

Section: Introductionmentioning

confidence: 99%

Modeling Performance of Hadoop Applications: A Journey from Queueing Networks to Stochastic Well Formed Nets

Ardagna

Bernardi

Gianniti

et al. 2016

Algorithms and Architectures for Parallel Processing

View full text Add to dashboard Cite

Nowadays, many enterprises commit to the extraction of actionable knowledge from huge datasets as part of their core business activities. Applications belong to very different domains such as fraud detection or one-to-one marketing, and encompass business analytics and support to decision making in both private and public sectors. In these scenarios, a central place is held by the MapReduce framework and in particular its open source implementation, Apache Hadoop. In such environments, new challenges arise in the area of jobs performance prediction, with the needs to provide Service Level Agreement guarantees to the end-user and to avoid waste of computational resources. In this paper we provide performance analysis models to estimate MapReduce job execution times in * Acknowledgments: This work has received funding from the European Union Horizon 2020 research and innovation program under grant agreement No. 644869 (DICE). Experimental data are available as open data at https://zenodo.org/record/58847#.V5i0wmXA45Q. 1Hadoop clusters governed by the YARN Capacity Scheduler. We propose models of increasing complexity and accuracy, ranging from queueing networks to stochastic well formed nets, able to estimate job performance under a number of scenarios of interest, including also unreliable resources. The accuracy of our models is evaluated by considering the TPC-DS industry benchmark running experiments on Amazon EC2 and the CINECA Italian supercomputing center. The results have shown that the average accuracy we can achieve is in the range 9-14%.

show abstract

“…What is the asymptotic behavior of the log weight sum log (X (n) ) dependent on the problem/instance size n? As remarked above the measurements X (n) of a particular MapReduce Sets for Replicated Join Expression assignment instance depend on the value n. In analogy to information intention [7] we assume that the log weight sums converge according to an asymptotic equipartition property, i.e.,…”

Section: Typicality Of Instancesmentioning

confidence: 99%