Exploiting mean field analysis to model performances of big data architectures

Castiglione, Aniello; Gribaudo, Marco; Iacono, Mauro; Palmieri, Francesco

doi:10.1016/j.future.2013.07.016

Cited by 57 publications

(30 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, Castiglione et al [12] describe a Big Data architecture based on Hadoop by means of stochastic PNs and apply Mean Field Analysis to obtain average metrics and estimate its performance. Another approach, presented by Barbierato et al [8], exploits Generalized Stochastic Petri Nets alongside other formalisms such as process algebras or Markov chains to develop multi-formalism models and capture HQL queries.…”

Section: Related Workmentioning

confidence: 99%

Modeling Performance of Hadoop Applications: A Journey from Queueing Networks to Stochastic Well Formed Nets

Ardagna

Bernardi

Gianniti

et al. 2016

Algorithms and Architectures for Parallel Processing

View full text Add to dashboard Cite

Nowadays, many enterprises commit to the extraction of actionable knowledge from huge datasets as part of their core business activities. Applications belong to very different domains such as fraud detection or one-to-one marketing, and encompass business analytics and support to decision making in both private and public sectors. In these scenarios, a central place is held by the MapReduce framework and in particular its open source implementation, Apache Hadoop. In such environments, new challenges arise in the area of jobs performance prediction, with the needs to provide Service Level Agreement guarantees to the end-user and to avoid waste of computational resources. In this paper we provide performance analysis models to estimate MapReduce job execution times in * Acknowledgments: This work has received funding from the European Union Horizon 2020 research and innovation program under grant agreement No. 644869 (DICE). Experimental data are available as open data at https://zenodo.org/record/58847#.V5i0wmXA45Q. 1Hadoop clusters governed by the YARN Capacity Scheduler. We propose models of increasing complexity and accuracy, ranging from queueing networks to stochastic well formed nets, able to estimate job performance under a number of scenarios of interest, including also unreliable resources. The accuracy of our models is evaluated by considering the TPC-DS industry benchmark running experiments on Amazon EC2 and the CINECA Italian supercomputing center. The results have shown that the average accuracy we can achieve is in the range 9-14%.

show abstract

Section: Related Workmentioning

confidence: 99%

Modeling Performance of Hadoop Applications: A Journey from Queueing Networks to Stochastic Well Formed Nets

Ardagna

Bernardi

Gianniti

et al. 2016

Algorithms and Architectures for Parallel Processing

View full text Add to dashboard Cite

show abstract

“…In order to enable administrators and developers to maintain the growth rate of the data, these systems' reliability assessment is critical. A set of methods for approximate inference of probabilistic models, based on MFA, can solve the performance evaluation system problem of big data [8]. Through behavioral modeling to assess the performance of data structure, MFA can calculate the related basic performance in a limited time.…”

Section: Mfa (Mean Field Analysis)mentioning

confidence: 99%

Research of Big Data Based on the Views of Technology and Application

Mo¹,

Li²

2015

AJIBM

View full text Add to dashboard Cite

In the era of big data, large amounts of data affect our work, life and study, even national economic development. It provides a new way of thinking and approaches to analyze and solve problems, which gradually becomes a hot research. Based on describing the concept and characteristics of big data, this paper describes the development of technologies in big data analysis and storage and analyses the trends and different values in commercial applications, manufacturing, biomedical science and other applications. At last, the authors sum up the existent challenges of big data applications and put forward the view that we should deal with big data challenges correctly.

show abstract

“…For example, in [3] advanced simulation techniques have been used to consider large MapReduce applications, and the proposed approach has been extended in [4] to insert it in a multiformalism context in order address more complex applicative scenarios. Mean-field approach has instead been used in [8] to consider different types of BigData applications. In this work, the considered application is modeled by pool depletion systems where a given number of tasks must be completed by a set of computing resources.…”

Section: Introductionmentioning

confidence: 99%

Modeling Replication and Erasure Coding in Large Scale Distributed Storage Systems Based on CEPH

Manini

Gribaudo

Iacono

2016

Lecture Notes in Information Systems and Organisation

Self Cite

View full text Add to dashboard Cite

Abstract. Today's most of high performance computing applications use parallel programming paradigms to reach the desired efficiency objectives. In particular, they divide the problem into small elements that can be solved in parallel by as many computing devices as available. Some examples are Apache Spark, the evolution of Hadoop and mapreduce, GPGPU (General Purpose Graphical Processing Units) applications, many-core and multi-core embedded systems. In many cases this type of applications can be modeled by pool depletion systems, that is queuing models characterized by a set of parallel servers whose goal is to execute a predetermined number of jobs. Although the modeling paradigm is very simple, it suffers from state space explosion, and can be used to model systems with a limited degree of parallelism only. The main contribution provided by this work consists of presenting a fluid approximation approach capturing the main features of the considered pool depletion systems and solving the above mentioned issues.

show abstract

Exploiting mean field analysis to model performances of big data architectures

Cited by 57 publications

References 11 publications

Modeling Performance of Hadoop Applications: A Journey from Queueing Networks to Stochastic Well Formed Nets

Modeling Performance of Hadoop Applications: A Journey from Queueing Networks to Stochastic Well Formed Nets

Research of Big Data Based on the Views of Technology and Application

Modeling Replication and Erasure Coding in Large Scale Distributed Storage Systems Based on CEPH

Contact Info

Product

Resources

About