Autoscaling for Hadoop Clusters

Gandhi, Anshul; Thota, Sidhartha; Dube, Parijat; Kochut, Andrzej; Zhang, Li

doi:10.1109/ic2e.2016.11

Cited by 27 publications

(12 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Few works for auto-scaling in the cloud are dedicated to database querying. There are some works that proposed autoscaling solutions for databases in the cloud but they focused on specific technologies, for example, MongoDB (Huang et al, 2013) or Hadoop (Gandhi et al, 2016). These works made performances (not monetary) metrics in their proposals.…”

Section: Discussionmentioning

confidence: 99%

Resource auto-scaling for SQL-like queries in the cloud based on parallel reinforcement learning

Kandi

Yin

Hameurlain

2019

IJGUC

View full text Add to dashboard Cite

Cloud computing is a technology that provides on-demand services in which the number of assigned resources can be automatically adjusted. A key challenge is how to choose the right number of resources so that the overall monetary cost is minimised. This problem, known as auto-scaling, was addressed in some existing works but most of them are dedicated to web applications. In these applications, it is assumed that the queries are atomic and each of them uses a single resource for a short period of time. However, this assumption cannot be considered for database applications. A query, in this case, contains many dependent and long tasks so several resources are required. We propose in this work an auto-scaling method based on reinforcement learning. The method is coupled with placement-scheduling. In the experimental section, we show the advantage of coupling the auto-scaling to the placement-scheduling by comparing our work to an existing auto-scaling method.Reference to this paper should be made as follows: Kandi, M.M., Yin, S. and Hameurlain, A. (2019) 'Resource auto-scaling for SQL-like queries in the cloud based on parallel reinforcement learning', Int.

show abstract

Section: Discussionmentioning

confidence: 99%

Resource auto-scaling for SQL-like queries in the cloud based on parallel reinforcement learning

Kandi

Yin

Hameurlain

2019

IJGUC

View full text Add to dashboard Cite

show abstract

“…In a cloud computing environment, many techniques have been developed for providing runtime guarantees for database queries [12, 52, 53, 78] and MapReduce-like jobs [14, 16, 21, 24, 31, 56, 72, 73], e.g., by estimating the number of computers that need to be allocated to a query/job for it to be likely to finish by a user-specified deadline. For the query/job, these techniques typically require using statistics collected from either its prior executions or first running it on many sample instances of the input data, do not continuously refine its estimated execution cost, and are not specifically designed and suitable for machine learning model building and data mining algorithm execution.…”

Section: Related Workmentioning

confidence: 99%

Toward a Progress Indicator for Machine Learning Model Building and Data Mining Algorithm Execution

Luo

2017

SIGKDD Explor. Newsl.

View full text Add to dashboard Cite

For user-friendliness, many software systems offer progress indicators for long-duration tasks. A typical progress indicator continuously estimates the remaining task execution time as well as the portion of the task that has been finished. Building a machine learning model often takes a long time, but no existing machine learning software supplies a non-trivial progress indicator. Similarly, running a data mining algorithm often takes a long time, but no existing data mining software provides a nontrivial progress indicator. In this article, we consider the problem of offering progress indicators for machine learning model building and data mining algorithm execution. We discuss the goals and challenges intrinsic to this problem. Then we describe an initial framework for implementing such progress indicators and two advanced, potential uses of them, with the goal of inspiring future research on this topic.

show abstract

“…Gandhi et al [7] define a fine-grained model for Hadoop/MapReduce. The computation model is approximated by second-order polynomials and, thus, would not be accurate if the workload has a higher-order time complexity, such as deep learning.…”

Section: Related Workmentioning

confidence: 99%

Modeling Scalability of Distributed Machine Learning

Ulanov

Simanovsky

Marwah

2017

2017 IEEE 33rd International Conference on Data Engineering (ICDE)

View full text Add to dashboard Cite

Abstract-Present day machine learning is computationally intensive and processes large amounts of data. It is implemented in a distributed fashion in order to address these scalability issues. The work is parallelized across a number of computing nodes. It is usually hard to estimate in advance how many nodes to use for a particular workload. We propose a simple framework for estimating the scalability of distributed machine learning algorithms. We measure the scalability by means of the speedup an algorithm achieves with more nodes. We propose time complexity models for gradient descent and graphical model inference. We validate our models with experiments on deep learning training and belief propagation. This framework was used to study the scalability of machine learning algorithms in Apache Spark.

show abstract

Autoscaling for Hadoop Clusters

Cited by 27 publications

References 12 publications

Resource auto-scaling for SQL-like queries in the cloud based on parallel reinforcement learning

Resource auto-scaling for SQL-like queries in the cloud based on parallel reinforcement learning

Toward a Progress Indicator for Machine Learning Model Building and Data Mining Algorithm Execution

Modeling Scalability of Distributed Machine Learning

Contact Info

Product

Resources

About