Predicting performance of parallel computations

Mak, Vivienne; Lundstrom, Stephen F.

doi:10.1109/71.80155

Cited by 97 publications

(53 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Hence, the performance metrics of such networks must be computed by considering the Markov Chain underlying the QN, which represents the possible states of the system [24]. However, such approaches are not fit for Hadoop systems, since the state space grows exponentially with the number of tasks [13,27], in the order of thousands in realistic MapReduce jobs. For this reason, a number of approximation methods have been proposed.…”

Section: Modeling Hadoop 2x Applications Performancementioning

confidence: 99%

Modeling Performance of Hadoop Applications: A Journey from Queueing Networks to Stochastic Well Formed Nets

Ardagna

Bernardi

Gianniti

et al. 2016

Algorithms and Architectures for Parallel Processing

View full text Add to dashboard Cite

Nowadays, many enterprises commit to the extraction of actionable knowledge from huge datasets as part of their core business activities. Applications belong to very different domains such as fraud detection or one-to-one marketing, and encompass business analytics and support to decision making in both private and public sectors. In these scenarios, a central place is held by the MapReduce framework and in particular its open source implementation, Apache Hadoop. In such environments, new challenges arise in the area of jobs performance prediction, with the needs to provide Service Level Agreement guarantees to the end-user and to avoid waste of computational resources. In this paper we provide performance analysis models to estimate MapReduce job execution times in * Acknowledgments: This work has received funding from the European Union Horizon 2020 research and innovation program under grant agreement No. 644869 (DICE). Experimental data are available as open data at https://zenodo.org/record/58847#.V5i0wmXA45Q. 1Hadoop clusters governed by the YARN Capacity Scheduler. We propose models of increasing complexity and accuracy, ranging from queueing networks to stochastic well formed nets, able to estimate job performance under a number of scenarios of interest, including also unreliable resources. The accuracy of our models is evaluated by considering the TPC-DS industry benchmark running experiments on Amazon EC2 and the CINECA Italian supercomputing center. The results have shown that the average accuracy we can achieve is in the range 9-14%.

show abstract

Section: Modeling Hadoop 2x Applications Performancementioning

confidence: 99%

Modeling Performance of Hadoop Applications: A Journey from Queueing Networks to Stochastic Well Formed Nets

Ardagna

Bernardi

Gianniti

et al. 2016

Algorithms and Architectures for Parallel Processing

View full text Add to dashboard Cite

show abstract

“…That said, it is possible to mitigate the issue by using a special kind of structure, as shown in [16] which considers the Markov Chain underlying the QN representing the possible states of the system. Unfortunately, as noted in [9,19], the state space grows exponentially when the tasks number corresponds to realistic MapReduce jobs-in the order of thousands-thus making the above approaches unsuitable.…”

Section: Related Workmentioning

confidence: 99%

Fluid Petri Nets for the Performance Evaluation of MapReduce Applications

Gianniti

Rizzi

Barbierato

et al. 2017

Proceedings of the 10th EAI International Conference on Performance Evaluation Methodologies and Tools

View full text Add to dashboard Cite

Big Data applications allow to successfully analyze large amounts of data not necessarily structured, though at the same time they present new challenges. For example, predicting the performance of frameworks such as Hadoop can be a costly task, hence the necessity to provide models that can be a valuable support for designers and developers. This paper provides a new contribution in studying a novel modeling approach based on fluid Petri nets to predict MapReduce jobs execution time.The experiments we performed at CINECA, the Italian supercomputing center, have shown that the achieved accuracy is within 16% of the actual measurements on average.

show abstract

“…Also, [24] reports the use of a hierarchical approach that combines Markov models and TGs. In [14], this combination is applied to the performance prediction of TGs executing in shared-memory multiprocessor environments. Stochastic Petri Nets (SPN) are also used to represent parallel programs.…”

Section: Preliminaries Overflow [3]mentioning

confidence: 99%

Task assignment heuristics for parallel and distributed CFD applications

Lopez-Benitez

Djomehri

Biswas

2007

IJCSE

View full text Add to dashboard Cite

Abstract. This paper proposes a task graph (TG) model to represent a single discrete step of multi-block overset grid computational fluid dynamics (CFD) applications. The T G model is then used to not only balance the computational workload across the overset grids but also t o reduce inter-grid communication costs. We have developed a set of task assignment heuristics based on the constraints inherent in this class of CFD problems. Two basic assignments, the smallest task first (STF) and the largest task first (LTF), are first presented. They are then systematically costs. To predict the performance of the proposed task assignment heuristics, extensive performance evaluations are conducted on a synthetic TG with tasks defined in terms of the number of grid points in predetermined overlapping grids. A TG derived from a realistic problem with eight million grid points is also used as a test case.

show abstract

Predicting performance of parallel computations

Cited by 97 publications

References 13 publications

Modeling Performance of Hadoop Applications: A Journey from Queueing Networks to Stochastic Well Formed Nets

Modeling Performance of Hadoop Applications: A Journey from Queueing Networks to Stochastic Well Formed Nets

Fluid Petri Nets for the Performance Evaluation of MapReduce Applications

Task assignment heuristics for parallel and distributed CFD applications

Contact Info

Product

Resources

About