No one (cluster) size fits all

Herodotou, Herodotos; Dong, Fei; Babu, Shivnath

doi:10.1145/2038916.2038934

Cited by 180 publications

(7 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, Starfish collects some key runtime information about the applications execution with the aim of generating meaningful application profiles; such profiles are in turn the basic elements to be exploited for Hadoop automatic configuration processes. Furthermore, also the cluster sizing problem has been tackled and successfully solved exploiting the same tool [21].…”

Section: Related Workmentioning

confidence: 99%

D-SPACE4Cloud: A Design Tool for Big Data Applications

Ciavotta

Gianniti

Ardagna

2016

Algorithms and Architectures for Parallel Processing

View full text Add to dashboard Cite

The last years have seen a steep rise in data generation worldwide, with the development and widespread adoption of several software projects targeting the Big Data paradigm. Many companies currently engage in Big Data analytics as part of their core business activities, nonetheless there are no tools and techniques to support the design of the underlying hardware configuration backing such systems. In particular, the focus in this report is set on Cloud deployed clusters, which represent a cost-effective alternative to on premises installations. We propose a novel tool implementing a battery of optimization and prediction techniques integrated so as to efficiently assess several alternative resource configurations, in order to determine the minimum cost cluster deployment satisfying Quality of Service constraints. Further, the experimental campaign conducted on real systems shows the validity and relevance of the proposed method.

show abstract

Section: Related Workmentioning

confidence: 99%

D-SPACE4Cloud: A Design Tool for Big Data Applications

Ciavotta

Gianniti

Ardagna

2016

Algorithms and Architectures for Parallel Processing

View full text Add to dashboard Cite

show abstract

“…Prior works report that there is not a one-size-fits-all VM type that is best for all workloads. 1,2,6,7 Thus, finding and matching the best cloud provider and best VM configuration to cost-efficiently run a workload became an important problem that has been approached by many authors. 2,3,[7][8][9][10] Moreover, the cloud infrastructure is dynamic 11 and can have a high variation in performance, 12 mostly because of the concurrent use of the physical resources from different VMs and users.…”

Section: Related Workmentioning

confidence: 99%

“…Each of these instantiations and providers results in different costs and system performances. Prior works report that there is not a one‐size‐fits‐all VM type that is best for all workloads 1,2,6,7 . Thus, finding and matching the best cloud provider and best VM configuration to cost‐efficiently run a workload became an important problem that has been approached by many authors 2,3,7‐10 …”

Section: Related Workmentioning

confidence: 99%

PB³Opt: Profile‐based biased Bayesian optimization to select computing clusters on the cloud

Camacho

Rosário

Napoli

et al. 2022

Concurrency and Computation

View full text Add to dashboard Cite

Given the wide variety of cloud computing resources for creating high-performance computer clusters and their complex performance relationship with applications, finding the optimal, or near-optimal, cluster is a complex problem. As a result, several approaches have been proposed to find the optimal, or near-optimal, cluster for a given high-performance computing workload, while reducing the search cost.Among the approaches found in the literature, Bayesian optimization is one of the most known and applied. However, it is still possible to increase its performance by integrating it with historical data related to workload behavior. In this context, we suggest the PB 3 Opt approach, which introduces a bias in the Bayesian optimization expected improvement acquisition function. The new acquisition function uses the ranking of computer clusters of previously explored workloads that have the same behavior as the workload being optimized. Our experimental results show that PB 3 Opt classifies the behavior of workloads in groups so that the average-ranking has 88.7% similarity with the ranking of the workload. With this, PB 3 Opt finds, for almost 95% of workloads, a solution that is less than or equal to 1.2 × worse than the optimal computer cluster. In addition, the PB 3 Opt works well when combined with the paramount iterations technique and is capable of reducing the search cost significantly.

show abstract

“…Meanwhile, at the high level, many of the prior research works on Hadoop MapReduce resource provisioning to accomplish certain application performance goals through dynamic job profiling, as in Babu [13], cluster size elasticizing, as in Herodotou et al [14], scaling past execution results to meet given Service Level Objectives (SLO), as in Verma et al [15], and matching an application's resource consumption with a database of similar signatures of other applications, as in Kambatla et al [16], could still be referenced when applying the BToP method in Nghiem and Figueira [5] to Apache Spark. This paper makes the following contributions:…”

Section: Introductionmentioning

confidence: 99%

Best Trade-Off Point Method for Efficient Resource Provisioning in Spark

Nghiem

2018

Algorithms

View full text Add to dashboard Cite

Considering the recent exponential growth in the amount of information processed in Big Data, the high energy consumed by data processing engines in datacenters has become a major issue, underlining the need for efficient resource allocation for more energy-efficient computing. We previously proposed the Best Trade-off Point (BToP) method, which provides a general approach and techniques based on an algorithm with mathematical formulas to find the best trade-off point on an elbow curve of performance vs. resources for efficient resource provisioning in Hadoop MapReduce. The BToP method is expected to work for any application or system which relies on a trade-off elbow curve, non-inverted or inverted, for making good decisions. In this paper, we apply the BToP method to the emerging cluster computing framework, Apache Spark, and show that its performance and energy consumption are better than Spark with its built-in dynamic resource allocation enabled. Our Spark-Bench tests confirm the effectiveness of using the BToP method with Spark to determine the optimal number of executors for any workload in production environments where job profiling for behavioral replication will lead to the most efficient resource provisioning.

show abstract

No one (cluster) size fits all

Cited by 180 publications

References 15 publications

D-SPACE4Cloud: A Design Tool for Big Data Applications

D-SPACE4Cloud: A Design Tool for Big Data Applications

PB³Opt: Profile‐based biased Bayesian optimization to select computing clusters on the cloud

Best Trade-Off Point Method for Efficient Resource Provisioning in Spark

Contact Info

Product

Resources

About

No one (cluster) size fits all

Cited by 180 publications

References 15 publications

D-SPACE4Cloud: A Design Tool for Big Data Applications

D-SPACE4Cloud: A Design Tool for Big Data Applications

PB3Opt: Profile‐based biased Bayesian optimization to select computing clusters on the cloud

Best Trade-Off Point Method for Efficient Resource Provisioning in Spark

Contact Info

Product

Resources

About

PB³Opt: Profile‐based biased Bayesian optimization to select computing clusters on the cloud