A Batch System with Fair Scheduling for Evolving Applications

Prabhakaran, Suraj; Iqbal, Mohsin; Rinke, Sebastian; Windisch, Christian; Wolf, Felix

doi:10.1109/icpp.2014.44

Cited by 17 publications

(10 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, dynamic scheduling work such as [29,30] can complete our approach by enabling the extension of our solutions to more traditional systems without burst bu↵ers. Dynamic scheduling extends a batch system with dynamic allocation facilities to support on-the-fly resource allocation to elastic jobs; although dynamic scheduling does not currently target any non-traditional constraints, including I/O bandwidth.…”

Section: Related Workmentioning

confidence: 98%

Scalable I/O-Aware Job Scheduling for Burst Buffer Enabled HPC Clusters

Herbein

Ahn

Lipari

et al. 2016

Proceedings of the 25th ACM International Symposium on High-Performance Parallel and Distributed Computing

View full text Add to dashboard Cite

The economics of flash vs. disk storage is driving HPC centers to incorporate faster solid-state burst bu↵ers into the storage hierarchy in exchange for smaller parallel file system (PFS) bandwidth. In systems with an underprovisioned PFS, avoiding I/O contention at the PFS level will become crucial to achieving high computational e ciency. In this paper, we propose novel batch job scheduling techniques that reduce such contention by integrating I/O awareness into scheduling policies such as EASY backfilling. We model the available bandwidth of links between each level of the storage hierarchy (i.e., burst bu↵ers, I/O network, and PFS), and our I/O-aware schedulers use this model to avoid contention at any level in the hierarchy. We integrate our approach into Flux, a next-generation resource and job management framework, and evaluate the e↵ectiveness and computational costs of our I/O-aware scheduling. Our results show that by reducing I/O contention for underprovisioned PFSes, our solution reduces job performance variability by up to 33% and decreases I/O-related utilization losses by up to 21%, which ultimately increases the amount of science performed by scientific workloads.

show abstract

Section: Related Workmentioning

confidence: 98%

Scalable I/O-Aware Job Scheduling for Burst Buffer Enabled HPC Clusters

Herbein

Ahn

Lipari

et al. 2016

Proceedings of the 25th ACM International Symposium on High-Performance Parallel and Distributed Computing

View full text Add to dashboard Cite

show abstract

“…Some grid scheduling systems offer support to these type of applications [20]. HPC schedulers have limited support for evolving jobs [30] and it is based on advance resource reservations. Our model proposal for HPC offers a possibility of freeing resources from dynamically malleable applications to serve evolving jobs or other rigid jobs with higher priority without using advance reservations.…”

Section: Related Workmentioning

confidence: 99%

A2l2

Álvarez

Östberg

Elmroth

et al. 2015

Proceedings of the 8th International Workshop on Virtualization Technologies in Distributed Computing

View full text Add to dashboard Cite

High-performance computing (HPC) is focused on providing large-scale compute capacity to scientific applications. HPC schedulers tend to be optimized for large parallel batch jobs and, as such, often overlook the requirements of other scientific applications. In this work, we propose a cloud-inspired HPC scheduling model that aims to capture application performance and requirement models (Application Aware -A2) and dynamically resize malleable application resource allocations to be able to support applications with critical performance or deadline requirements. (Low Latency allocation -L2). The proposed model incorporates measures to improve data-intensive applications performance on HPC systems and is derived from a set of cloud scheduling techniques that are identified as applicable in HPC environments. The model places special focus on dynamically malleable applications; data-intensive applications that support dynamic resource allocation without incurring severe performance penalties; which are proposed for fine-grained backfilling and dynamic resource allocation control without job preemption.

show abstract

“…Evolving requests are now scheduled, which may steal resources from the StartNow jobs, thereby causing delay to the StartNow as well as to the StartLater jobs. In our previous work, we designed dynamic fairness policies for the Maui scheduler to ensure fairness between such unpredictably evolving requests and static requests, which were based on administratorconfigurable parameters [11]. With these policies, it was shown that evolving jobs can be served (as long as idle nodes are available) with fairness by controlled and admissible delay that can also take historic delays for users into consideration.…”

Section: Malleable Scheduling With Mauimentioning

confidence: 99%

“…In our previous work [11], we extended the Maui scheduler to support evolving applications. To maintain fairness between evolving requests and static requests (jobs queued through qsub), new dynamic fairness policies were introduced.…”

Section: Malleable Scheduling With Mauimentioning

confidence: 99%

“…To this end, this paper advances the state of the art in batch job management by proposing an extended Torque/Maui batch system with a novel and an efficient scheduling and resource management strategy for malleable jobs. By combining it with our previous work on supporting unpredictably evolving applications [11], we present the first production batch system for the combined scheduling of rigid, malleable, and evolving jobs unlike past works which either simulated malleable jobs [12], [13] or demonstrated prototype/demo schedulers [14], [15], [16]. Towards establishing a next generation batch system and a tight coupling with parallel runtime, our main contributions are:…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Batch System with Efficient Adaptive Scheduling for Malleable and Evolving Applications

Prabhakaran

Neumann

Rinke

et al. 2015

2015 IEEE International Parallel and Distributed Processing Symposium

Self Cite

View full text Add to dashboard Cite

Abstract-The throughput of supercomputers depends not only on efficient job scheduling but also on the type of jobs that form the workload. Malleable jobs are most favorable for a cluster as they can dynamically adapt to a changing allocation of resources. The batch system can expand or shrink a running malleable job to improve system utilization, throughput, and response times. In the past, however, the rigid nature of commonly used programming models like MPI made writing malleable applications a daunting task, which is why it remained largely unrealized. This is now changing. To improve fault tolerance, load imbalance, and energy efficiency in emerging exascale systems, more adaptive programming paradigms such as Charm++ enter the scene. Although they offer better support for malleability, current batch systems still lack management facilities for malleable jobs and are therefore incapable of leveraging their potential. In this paper, we present an extension of the Torque/Maui batch system for malleability. We propose a novel malleable job scheduling strategy and show the first batch system capable of efficiently managing rigid, malleable, and evolving jobs together. We demonstrate that our strategy achieves consistently superior performance in comparison to every other state-of-the-art malleable job scheduling strategy under varying dynamics of the workload.

show abstract

A Batch System with Fair Scheduling for Evolving Applications

Cited by 17 publications

References 17 publications

Scalable I/O-Aware Job Scheduling for Burst Buffer Enabled HPC Clusters

Scalable I/O-Aware Job Scheduling for Burst Buffer Enabled HPC Clusters

A2l2

A Batch System with Efficient Adaptive Scheduling for Malleable and Evolving Applications

Contact Info

Product

Resources

About