Elasecutor: Elastic Executor Scheduling in Data Analytics Systems

Liu, Libin; Xu, Hong

doi:10.1109/tnet.2021.3050927

Cited by 8 publications

(4 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Constraint ( 9) and ( 10) ensure that minimum resource is allocated if a job is still training. Constraint (11) let node size return back to zero after a job has done. Constraints ( 12)∼( 14) are the bounds of decision variables.…”

Section: Non-linear Formulationmentioning

confidence: 99%

“…We must linearize the non-linear constraints and simplify the above model formulation to make it solvable and fast. First, we drop the binary variables y t i and replace constraints ( 7)- (11) with a new constraint (15). This constraint allows nodes not return back to zero after a job has finished training.…”

Section: Linearized and Simplified Formulationmentioning

confidence: 99%

“…Its objective is to reduce energy consumption of MapReduce jobs. Liu and Xu [11] proposed an executor scheduler that is able to dynamically allocate and size resources to Spark jobs in order to minimize resource fragmentation. Another work by Javadi et al [12] also present a workload resource scheduler for Spark jobs that can significantly increase cloud resource usage.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An Optimal Resource Allocator of Elastic Training for Deep Learning Jobs on Cloud

Liang¹,

Zhu²,

Zhou³

et al. 2021

Preprint

View full text Add to dashboard Cite

Cloud training platforms, such as Amazon Web Services and Huawei Cloud provide users with computational resources to train their deep learning jobs. Elastic training is a service embedded in cloud training platforms that dynamically scales up or down the resources allocated to a job. The core technique of an elastic training system is to best allocate limited resources among heterogeneous jobs in terms of shorter queueing delay and higher training efficiency. This paper presents an optimal resource allocator for elastic training system that leverages a mixed-integer programming (MIP) model to maximize the training progress of deep learning jobs. We take advantage of the real-world job data obtained from ModelArts, the deep learning training platform of Huawei Cloud and conduct simulation experiments to compare the optimal resource allocator with a greedy one as benchmark. Numerical results show that the proposed allocator can reduce queuing time by up to 32% and accelerate training efficiency by up to 24% relative to the greedy resource allocator, thereby greatly improving user experience with Huawei ModelArts and potentially enabling the realization of higher profits for the product. Also, the optimal resource allocator is fast in decision-making, taking merely 0.4 seconds on average.

show abstract

Section: Non-linear Formulationmentioning

confidence: 99%

Section: Linearized and Simplified Formulationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

An Optimal Resource Allocator of Elastic Training for Deep Learning Jobs on Cloud

Liang¹,

Zhu²,

Zhou³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…CherryPick [27] leverages Bayesian Optimization to build performance models for various applications. Elasecutor [32] relies on support vector regression (SVR) to train the model that predicts demand time series. When configuring large-scale systems with complex parameters, it is not wise to rely just on empiricism.…”

Section: Related Workmentioning

confidence: 99%

Adaptive Online Estimation of Thrashing-Avoiding Memory Reservations for Long-Lived Containers

Lin

Liu

Cai

et al. 2021

Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

View full text Add to dashboard Cite

Data-intensive computing systems in cloud datacenters create longlived containers and allocate memory resource for them to execute long-running applications. It is a challenge to exactly estimate how much memory should be reserved for containers to enable smooth application execution and high resource utilization as well. Current state-of-the-art work has two limitations. First, prediction accuracy is restricted by the monotonicity of the iterative search. Second, application performance fluctuates due to the termination conditions. In this paper, we propose two improved strategies based on MEER, called MEER+ and Deep-MEER, which are designed to assist in memory allocation upon resource manager like YARN. MEER+ has one more step of approximation than MEER, to make the iterative search bi-directional and better approach the optimal value. Based on reinforcement learning and rich data, Deep-MEER achieves thrashingavoiding estimation without involving termination conditions. Based on the different input requirements and advantages, a scheme to adaptively adopt MEER+ and Deep-MEER in cluster life cycle is proposed. We have evaluated MEER+ and Deep-MEER. Our experimental results show that MEER+ and Deep-MEER yield up to 88% and 20% higher accuracy. Moreover, Deep-MEER guarantees stable performance for applications during recurring executions.

show abstract

Energy-aware scheduling for spark job based on deep reinforcement learning in cloud

Shi

et al. 2023

Computing

View full text Add to dashboard Cite

Elasecutor: Elastic Executor Scheduling in Data Analytics Systems

Cited by 8 publications

References 38 publications

An Optimal Resource Allocator of Elastic Training for Deep Learning Jobs on Cloud

An Optimal Resource Allocator of Elastic Training for Deep Learning Jobs on Cloud

Adaptive Online Estimation of Thrashing-Avoiding Memory Reservations for Long-Lived Containers

Energy-aware scheduling for spark job based on deep reinforcement learning in cloud

Contact Info

Product

Resources

About