RubberBand

Abstract

Hyperparameter tuning is essential to achieving state-of-theart accuracy in machine learning (ML), but requires substantial compute resources to perform. Existing systems primarily focus on eectively allocating resources for a hyperparameter tuning job under xed resource constraints. We show that the available parallelism in such jobs changes dynamically over the course of execution and, therefore, presents an opportunity to leverage the elasticity of the cloud.In particular, we address the problem of minimizing the nancial cost of executing a hyperparameter tuning job, subject to a time constraint. We present RubberBand-the rst framework for cost-ecient, elastic execution of hyperparameter tuning jobs in the cloud. RubberBand utilizes performance instrumentation and cloud pricing to model job completion time and cost prior to runtime, and generate a cost-ecient, elastic resource allocation plan. RubberBand is able to eciently execute this plan and realize a cost reduction of up to 2x in comparison to static allocation baselines.CCS Concepts • Computing methodologies → Distributed computing methodologies; Machine learning.

Detection of Leaf Spot Disease in Oil Palm Seedlings Using Convolutional Neural Network Method

“…By repeating and varying the configuration, it is expected that the model can be optimally adjusted so that it can provide accurate predictions on new data. Through the hyperparameter tuning process, the model can be optimised and able to produce better performance than using the default configuration [18].…”

Building efficient and practical machine learning systems

“…Hyperparameter Tuning (i.e., Hyperparameter Optimization, HPO) aims to identify the optimal hyperparameters via massive configuration exploration [112,118]. In the general workflow of an HPO job: (1) the user designates a search space of hyperparameters to explore; (2) the tuning algorithm creates a set of training trials and each trial contains one unique hyperparameter configuration sampled from the search space;…”

“…Existing research works typically optimize HPO efficiency from the tuning algorithm [132][133][134][135][136][137][138][139][140] or system [7,32,112,118,119,127,141,142] aspects:…”

“…Due to the high dimensionality of the search space, a hyperparameter tuning job typically contains a large group of trials, each with a unique configuration [32]. To accelerate the tuning process, tech companies and researchers build hyperparameter tuning systems as cloud services [3,6,116,117] or standalone frameworks [7,10,32,112,118,119] (Table 3.1). However, we argue that state-of-the-art tuning systems are still expensive and inefficient in practice, as they suffer from several fundamental problems:…”

“…Moreover, despite the growing trend of foundation models being employed in clusters, large-scale models often fail to fully utilize hardware resources due to the huge communication overhead and the presence of bubbles in the pipeline parallelism [126]. To improve resource utilization, some novel tuning systems incorporate features such as elastic training [112,118,119], GPU sharing [32], and inter-trial fusion [127]. However, these systems have certain limitations ( §3.7) and often require substantial resources to explore trivial trials, which results in limited resources being contributed to the final model.…”