Optimal Capacity Allocation for Executing MapReduce Jobs in Cloud Systems

Malekimajd, Marzieh; Rizzi, Alessandro; Ardagna, Danilo; Ciavotta, Michele; Passacantando, Mauro; Movaghar, Ali

doi:10.1109/synasc.2014.58

Cited by 10 publications

(16 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We remark that now constraints (14) are convex (the proof is reported in [18]). The convexity of all the constraints of problem (P1) allows to prove the following result.…”

Section: Optimization Problemmentioning

confidence: 95%

“…Such results are used in the remaining sections to define the constraints of the CA problem that guarantee job deadlines are met. For space limitation, some proofs are omitted and reported in [18].…”

Section: Estimating Job Execution Times In Shared Clustersmentioning

confidence: 99%

“…Problem (P0) has a linear objective function but constraints (6) are non-linear and non-convex (the proof is reported in [18]). To overcome the non-convexity of the constraints, we introduce new decision variables Ψ i = 1/h i , for any i ∈ U, to replace h i .…”

Section: Optimization Problemmentioning

confidence: 99%

“…The proof of Theorem 4.1 is reported in [18]. The results of Theorem 4.1 allow to transform (P1) into an equivalent linear programming problem, which can be solved very quickly by state of the art solvers.…”

Section: Optimization Problemmentioning

confidence: 99%

“…The proof of Theorem 4.2 is reported in [18]. Since (P2) is a linear problem, commercial and open source solvers currently available are able to solve efficiently very large instances.…”

Section: Theorem 42 (P1) Is Equivalent To the Following Problemmentioning

confidence: 99%

See 4 more Smart Citations

Optimal Map Reduce Job Capacity Allocation in Cloud Systems

Malekimajd

Ardagna

Ciavotta

et al. 2015

SIGMETRICS Perform. Eval. Rev.

View full text Add to dashboard Cite

We are entering a Big Data world. Many sectors of our economy are now guided by data-driven decision processes. Big Data and business intelligence applications are facilitated by the MapReduce programming model while, at infrastructural layer, cloud computing provides flexible and cost effective solutions for allocating on demand large clusters. Capacity allocation in such systems is a key challenge to provide performance for MapReduce jobs and minimize cloud resource costs. The contribution of this paper is twofold: (i) we provide new upper and lower bounds for MapReduce job execution time in shared Hadoop clusters, (ii) we formulate a linear programming model able to minimize cloud resources costs and job rejection penalties for the execution of jobs of multiple classes with (soft) deadline guarantees. Simulation results show how the execution time of MapReduce jobs falls within 14% of our upper bound on average. Moreover, numerical analyses demonstrate that our method is able to determine the global optimal solution of the linear problem for systems including up to 1,000 user classes in less than 0.5 seconds.

show abstract

“…We remark that now constraints (14) are convex (the proof is reported in [18]). The convexity of all the constraints of problem (P1) allows to prove the following result.…”

Section: Optimization Problemmentioning

confidence: 95%

Section: Estimating Job Execution Times In Shared Clustersmentioning

confidence: 99%

Section: Optimization Problemmentioning

confidence: 99%

Section: Optimization Problemmentioning

confidence: 99%

“…The proof of Theorem 4.2 is reported in [18]. Since (P2) is a linear problem, commercial and open source solvers currently available are able to solve efficiently very large instances.…”

Section: Theorem 42 (P1) Is Equivalent To the Following Problemmentioning

confidence: 99%

See 3 more Smart Citations

Optimal Map Reduce Job Capacity Allocation in Cloud Systems

Malekimajd

Ardagna

Ciavotta

et al. 2015

SIGMETRICS Perform. Eval. Rev.

View full text Add to dashboard Cite

show abstract

D-SPACE4Cloud: A Design Tool for Big Data Applications

Ciavotta

Gianniti

Ardagna

2016

Algorithms and Architectures for Parallel Processing

View full text Add to dashboard Cite

The last years have seen a steep rise in data generation worldwide, with the development and widespread adoption of several software projects targeting the Big Data paradigm. Many companies currently engage in Big Data analytics as part of their core business activities, nonetheless there are no tools and techniques to support the design of the underlying hardware configuration backing such systems. In particular, the focus in this report is set on Cloud deployed clusters, which represent a cost-effective alternative to on premises installations. We propose a novel tool implementing a battery of optimization and prediction techniques integrated so as to efficiently assess several alternative resource configurations, in order to determine the minimum cost cluster deployment satisfying Quality of Service constraints. Further, the experimental campaign conducted on real systems shows the validity and relevance of the proposed method.

show abstract

An optimization framework for the capacity allocation and admission control of MapReduce jobs in cloud systems

et al. 2018

Self Cite

View full text Add to dashboard Cite

It is safe to claim that we live in a Big Data world. In effect, many sectors of our economy are already guided by data-driven decision processes. Big Data and Business Intelligence applications are facilitated by the MapReduce programming model while, at infrastructural layer, cloud computing provides flexible and cost effective solutions to provide on demand large clusters. Capacity allocation in such systems, meant as the problem of providing computational power to support concurrent MapReduce applications in a cost effective fashion, represents a challenge of paramount importance. In this paper we lay the foundation for a solution implementing admission control and capacity allocation for MapReduce jobs with a priori deadline guarantees. In particular, shared Hadoop 2.x clusters supporting batch and/or interactive jobs are targeted. We formulate a linear programming model able The simulations and numerical analyses have been performed under the Windows Azure

show abstract

Optimal Capacity Allocation for Executing MapReduce Jobs in Cloud Systems

Cited by 10 publications

References 17 publications

Optimal Map Reduce Job Capacity Allocation in Cloud Systems

Optimal Map Reduce Job Capacity Allocation in Cloud Systems

D-SPACE4Cloud: A Design Tool for Big Data Applications

An optimization framework for the capacity allocation and admission control of MapReduce jobs in cloud systems

Contact Info

Product

Resources

About