A Job Scheduling Approach for Multi-core Clusters Based on Virtual Malleability

Utrera, Gladys; Tabik, Siham; Corbalan, Julita; Labarta, Jesús

doi:10.1007/978-3-642-32820-6_20

Cited by 18 publications

(15 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Utrera et al [12] propose a VM-based FCFS-malleable algorithm, which aims to improve system utilization and shorten waiting time. He et al [13] introduce a two-level framework for malleable job scheduling.…”

Section: Parallel Job Schedulingmentioning

confidence: 99%

See 1 more Smart Citation

HRF

Tuo

Jin

et al. 2015

Proceedings of the 12th ACM International Conference on Computing Frontiers

View full text Add to dashboard Cite

Moldable jobs, which allow the number of allocated processors to be adjusted before running in clusters, have attracted increasing concern in parallel job scheduling research. Compared with traditional rigid jobs where the number of allocated processors is fixed, moldable jobs are more flexible and therefore have more potential for improving their average turnaround time (a crucial metric to describe performance of jobs in a cluster). Average turnaround time of moldable jobs depends greatly on resource allocation schemes. Unfortunately, existing schemes do not perform well in reducing average turnaround time, either because they only consider a single job's turnaround time instead of the average turnaround time of all jobs, or because they just aim at fairness between short and long jobs instead of their average turnaround time. In this paper, we investigate how resource allocation affects the average turnaround time of moldable jobs in clusters, and propose a scheme named HRF (highest revenue first), which allocates processors according to the highest revenue of shortening runtime. In our simulations, experimental results show that HRF can reduce average turnaround time up to 71% when compared with state-of-the-art schemes.

show abstract

Section: Parallel Job Schedulingmentioning

confidence: 99%

“…Malleable jobs are more flexible than moldable jobs, but they require support, libraries and runtime systems [12], which are very costly in practice. Therefore in this paper, we study the scheduling of moldable jobs among the above-mentioned parallel jobs.…”

Section: Parallel Job Schedulingmentioning

confidence: 99%

HRF

Tuo

Jin

et al. 2015

Proceedings of the 12th ACM International Conference on Computing Frontiers

View full text Add to dashboard Cite

show abstract

“…Several studies have been done to reduce programming complexity when developing malleable applications. Utrera [35] uses folding techniques and a FCFS-malleable policy that uses co-scheduling and oversubscription to start MPI jobs when not enough resources are available. Kale [20] implement malleable and evolving jobs on the top of Charm++, and defined a scheduling policy based on equipartition.…”

Section: Related Workmentioning

confidence: 99%

Holistic Slowdown Driven Scheduling and Resource Management for Malleable Jobs

D'Amico

Jokanović

Corbalan

2019

Proceedings of the 48th International Conference on Parallel Processing

Self Cite

View full text Add to dashboard Cite

In job scheduling, the concept of malleability has been explored since many years ago. Research shows that malleability improves system performance, but its utilization in HPC never became widespread. The causes are the difficulty in developing malleable applications, and the lack of support and integration of the different layers of the HPC software stack. However, in the last years, malleability in job scheduling is becoming more critical because of the increasing complexity of hardware and workloads. In this context, using nodes in an exclusive mode is not always the most efficient solution as in traditional HPC jobs, where applications were highly tuned for static allocations, but offering zero flexibility to dynamic executions. This paper proposes a new holistic, dynamic job scheduling policy, Slowdown Driven (SD-Policy), which exploits the malleability of applications as the key technology to reduce the average slowdown and response time of jobs. SD-Policy is based on backfill and node sharing. It applies malleability to running jobs to make room for jobs that will run with a reduced set of resources, only when the estimated slowdown improves over the static approach. We implemented SD-Policy in SLURM and evaluated it in a real production environment, and with a simulator using workloads of up to 198K jobs. Results show better resource utilization with the reduction of makespan, response time, slowdown, and energy consumption, up to respectively 7%, 50%, 70%, and 6%, for the evaluated workloads. CCS CONCEPTS• Software and its engineering → Scheduling; • Computer systems organization → Multicore architectures.

show abstract

“…These resource changes can either increase (expand operation) or reduce (shrink operation) the number of processors. Malleable jobs have shown to significantly improve the performance of a batch system in terms of both system and user-centric metrics such as system utilization and average response time [7], [8]. Enabling malleable jobs in HPC systems requires an adaptive parallel runtime system and an adaptive batch system.…”

Section: Introductionmentioning

confidence: 99%

Extending SLURM for Dynamic Resource-Aware Adaptive Batch Scheduling

Chadha¹,

John²,

Gerndt³

2020

Preprint

View full text Add to dashboard Cite

With the growing constraints on power budget and increasing hardware failure rates, the operation of future exascale systems faces several challenges. Towards this, resource awareness and adaptivity by enabling malleable jobs has been actively researched in the HPC community. Malleable jobs can change their computing resources at runtime and can significantly improve HPC system performance. However, due to the rigid nature of popular parallel programming paradigms such as MPI and lack of support for dynamic resource management in batch systems, malleable jobs have been largely unrealized. In this paper, we extend the SLURM batch system to support the execution and batch scheduling of malleable jobs. The malleable applications are written using a new adaptive parallel paradigm called Invasive MPI which extends the MPI standard to support resource-adaptivity at runtime. We propose two malleable job scheduling strategies to support performance-aware and poweraware dynamic reconfiguration decisions at runtime. We implement the strategies in SLURM and evaluate them on a production HPC system. Results for our performance-aware scheduling strategy show improvements in makespan, average system utilization, average response, and waiting times as compared to other scheduling strategies. Moreover, we demonstrate dynamic power corridor management using our power-aware strategy.

show abstract

A Job Scheduling Approach for Multi-core Clusters Based on Virtual Malleability

Cited by 18 publications

References 20 publications

HRF

HRF

Holistic Slowdown Driven Scheduling and Resource Management for Malleable Jobs

Extending SLURM for Dynamic Resource-Aware Adaptive Batch Scheduling

Contact Info

Product

Resources

About