Performance under Failures of MapReduce Applications

Jin, Jin; Qiao, Qiao; Sun, Dong; Li, Qing

doi:10.1109/ccgrid.2011.84

Cited by 26 publications

(16 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They evaluate the correctness of the system and carry out a trade-off analysis of the number of resources versus processing time and resource cost with CPNTools [18]. Further works with PNs and MapReduce are oriented to measuring performance under failures [19] or studying the fault tolerance mechanism [28].…”

Section: Related Workmentioning

confidence: 99%

Modeling Performance of Hadoop Applications: A Journey from Queueing Networks to Stochastic Well Formed Nets

Ardagna

Bernardi

Gianniti

et al. 2016

Algorithms and Architectures for Parallel Processing

View full text Add to dashboard Cite

Nowadays, many enterprises commit to the extraction of actionable knowledge from huge datasets as part of their core business activities. Applications belong to very different domains such as fraud detection or one-to-one marketing, and encompass business analytics and support to decision making in both private and public sectors. In these scenarios, a central place is held by the MapReduce framework and in particular its open source implementation, Apache Hadoop. In such environments, new challenges arise in the area of jobs performance prediction, with the needs to provide Service Level Agreement guarantees to the end-user and to avoid waste of computational resources. In this paper we provide performance analysis models to estimate MapReduce job execution times in * Acknowledgments: This work has received funding from the European Union Horizon 2020 research and innovation program under grant agreement No. 644869 (DICE). Experimental data are available as open data at https://zenodo.org/record/58847#.V5i0wmXA45Q. 1Hadoop clusters governed by the YARN Capacity Scheduler. We propose models of increasing complexity and accuracy, ranging from queueing networks to stochastic well formed nets, able to estimate job performance under a number of scenarios of interest, including also unreliable resources. The accuracy of our models is evaluated by considering the TPC-DS industry benchmark running experiments on Amazon EC2 and the CINECA Italian supercomputing center. The results have shown that the average accuracy we can achieve is in the range 9-14%.

show abstract

Section: Related Workmentioning

confidence: 99%

Modeling Performance of Hadoop Applications: A Journey from Queueing Networks to Stochastic Well Formed Nets

Ardagna

Bernardi

Gianniti

et al. 2016

Algorithms and Architectures for Parallel Processing

View full text Add to dashboard Cite

show abstract

“…They claimed that a single failure led to variable and unpredictable job running times. Jin et al [18] derived a stochastic model to predict both the best and worst job completion time of MapReduce applications on node failures.…”

Section: Related Workmentioning

confidence: 99%

Analyzing job completion reliability and job energy consumption for a heterogeneous MapReduce cluster under different intermediate-data replication policies

2014

View full text Add to dashboard Cite

Recently, MapReduce has been a popular distributed programming framework for solving data-intensive applications. However, a large-scale MapReduce cluster has inevitable machine/node failures and considerable energy consumption. To solve these problems, MapReduce has employed several policies for replicating input data, storing/replicating intermediate data, and re-executing failed tasks. In this study, we concentrate on two typical policies for storing/replicating intermediate data, and derive the job completion reliability (JCR for short) and job energy consumption (JEC for short) of a MapReduce cluster when the two policies are individually employed. The two policies are further analyzed and compared given various scenarios in which jobs with different input data sizes, numbers of reduce tasks, and other parameters are run in a MapReduce cluster with two extreme parallel execution capabilities. From the analytical results, MapReduce managers are able to comprehend how the two policies influence the JCR and JEC of a MapReduce cluster.

show abstract

“…The conventional wisdom is to make a number of replicas to the primary copy [13,17]. When the primary copy is failed, one of the replicas will be restored to replace the failed primary copy.…”

Section: E Fault Tolerancementioning

confidence: 99%

HyCache+: Towards Scalable High-Performance Caching Middleware for Parallel File Systems

Zhao

Qiao

Raicu

2014

2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

Self Cite

View full text Add to dashboard Cite

Abstract-The ever-growing gap between the computation and I/O is one of the fundamental challenges for future computing systems. This computation-I/O gap is even larger for modern large scale high-performance systems due to their state-of-the-art yet decades long architecture: the compute and storage resources form two cliques that are interconnected with shared networking infrastructure. This paper presents a distributed storage middleware, called HyCache+, right on the compute nodes, which allows I/O to effectively leverage the high bi-section bandwidth of the high-speed interconnect of massively parallel high-end computing systems. HyCache+ provides the POSIX interface to end users with the memory-class I/O throughput and latency, and transparently swap the cached data with the existing slowspeed but high-capacity networked attached storage. HyCache+ has the potential to achieve both high performance and lowcost large capacity, the best of both worlds. To further improve the caching performance from the perspective of the global storage system, we propose a 2-phase mechanism to cache the hot data for parallel applications, called 2-Layer Scheduling (2LS), which minimizes the file size to be transferred between compute nodes and heuristically replaces files in the cache. We deploy HyCache+ on the IBM BlueGene/P supercomputer, and observe two orders of magnitude faster I/O throughput than the default GPFS parallel file system. Furthermore, the proposed heuristic caching approach shows 29X speedup over the traditional LRU algorithm.

show abstract

Performance under Failures of MapReduce Applications

Cited by 26 publications

References 4 publications

Modeling Performance of Hadoop Applications: A Journey from Queueing Networks to Stochastic Well Formed Nets

Modeling Performance of Hadoop Applications: A Journey from Queueing Networks to Stochastic Well Formed Nets

Analyzing job completion reliability and job energy consumption for a heterogeneous MapReduce cluster under different intermediate-data replication policies

HyCache+: Towards Scalable High-Performance Caching Middleware for Parallel File Systems

Contact Info

Product

Resources

About