Supporting Real-Time Jobs on the IBM Blue Gene/Q: Simulation-Based Study

Wang, Daihou; Jung, Eun-Sung; Kettimuthu, Rajkumar; Foster, Ian; Foran, David J.; Parashar, Manish

doi:10.1007/978-3-319-77398-8_5

Cited by 6 publications

(6 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The high priority queue contains all other RTJs; these jobs can preempt running BJs under certain conditions. We note that the high priority queue in this scheme is different from the high-priority queue scheme in [54]. The high priority queue in this scheme only has RTJs that have reached a slowdown threshold, while he high-priority queue scheme in [54] contains all RTJs.…”

Section: Heuristic Detailsmentioning

confidence: 93%

“…However, the production system such as Mira does not support such functions, and we, therefore, decided not to include malleability as basic techniques. This section is the summary of our previous work [54] to help understand the baseline techniques.…”

Section: Basic Scheduling Techniquesmentioning

confidence: 99%

“…As summarized in Section 4, our previous work [54] showed that simple as well as some sophisticated heuristics to support RTJs can be detrimental to certain classes of batch jobs. Here we build on that work to develop a new heuristic that addresses the issues with the heuristics described in Section 4.…”

Section: Novel Scheduling Heuristicmentioning

confidence: 99%

“…Here we build on that work to develop a new heuristic that addresses the issues with the heuristics described in Section 4. To develop our scheduling heuristic, we made the additional changes to the modified Cobalt scheduling heuristic in [54] as follows.…”

Section: Novel Scheduling Heuristicmentioning

confidence: 99%

See 3 more Smart Citations

Towards Accommodating Real-time Jobs on HPC Platforms

Nickolay¹,

Jung²,

Kettimuthu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Increasing data volumes in scientific experiments necessitate the use of high performance computing (HPC) resources for data analysis. In many scientific fields, the data generated from scientific instruments and supercomputer simulations must be analyzed rapidly. In fact, the requirement for quasiinstant feedback is growing. Scientists want to use results from one experiment to guide the selection of the next or even to improve the course of a single experiment. Current HPC systems are typically batch-scheduled under policies in which an arriving job is run immediately only if enough resources are available; otherwise it is queued. It is hard for these systems to support real-time jobs. Real-time jobs, in order to meet their requirements, should sometimes have to preempt batch jobs and/or be scheduled ahead of batch jobs that were submitted earlier. Accommodating real-time jobs may negatively impact system utilization also, especially when preemption/restart of batch jobs is involved. We first explore several existing scheduling strategies to make real-time jobs more likely to be scheduled in due time. We then rigorously formulate the problem as a mixedinteger linear programming for offline scheduling and develop novel scheduling heuristics for online scheduling. We perform simulation studies using trace logs of Mira, the IBM BG/Q system at Argonne National Laboratory, to quantify the impact of real-time jobs on batch job performance for various percentages of real-time jobs in the workload. We present new in-sights gained from grouping jobs into different categories based on runtime and the number of nodes used and studying the performance of each category. Our results show that with 10% real-time job percentages, just-in-time checkpointing combined with our heuristic can improve the slowdowns of real-time jobs by 35% while limiting the increase of the slowdowns of batch jobs to 10%.

show abstract

Section: Heuristic Detailsmentioning

confidence: 93%

Section: Basic Scheduling Techniquesmentioning

confidence: 99%

Section: Novel Scheduling Heuristicmentioning

confidence: 99%

Section: Novel Scheduling Heuristicmentioning

confidence: 99%

See 2 more Smart Citations

Towards Accommodating Real-time Jobs on HPC Platforms

Nickolay¹,

Jung²,

Kettimuthu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Scheduling of batch and on-demand jobs for concurrent execution where resources sharing is limited to each type of job has been widely studied [12]- [19] in the past. However, not much work has been done for collocating both batch and on-demand jobs on the same set of resources [20]. SPRUCE (Special Priority and Urgent Computing Environment) [21] supports on-demand jobs by considering a basic preemptive scheduling scheme with no checkpointing.…”

Section: Related Workmentioning

confidence: 99%

CoSim: A Simulator for Co-Scheduling of Batch and On-Demand Jobs in HPC Datacenters

Maurya

Nicolae

Guliani

et al. 2020

2020 IEEE/ACM 24th International Symposium on Distributed Simulation and Real Time Applications (DS-RT)

View full text Add to dashboard Cite

The increasing scale and complexity of scientific applications are rapidly transforming the ecosystem of tools, methods, and workflows adopted by the high-performance computing (HPC) community. Big data analytics and deep learning are gaining traction as essential components in this ecosystem in a variety of scenarios, such as, steering of experimental instruments, acceleration of high-fidelity simulations through surrogate computations, and guided ensemble searches. In this context, the batch job model traditionally adopted by the supercomputing infrastructures needs to be complemented with support to schedule opportunistic on-demand analytics jobs, leading to the problem of efficient preemption of batch jobs with minimum loss of progress. In this paper, we design and implement a simulator, CoSim, that enables on-the-fly analysis of the trade-offs arising between delaying the start of opportunistic on-demand jobs, which leads to longer analytics latency, and loss of progress due to preemption of batch jobs, which is necessary to make room for on-demand jobs. To this end, we propose an algorithm based on dynamic programming with predictable performance and scalability that enables supercomputing infrastructure schedulers to analyze the aforementioned trade-off and take decisions in near real-time. Compared with other state-of-art approaches using traces of the Theta pre-Exascale machine, our approach is capable of finding the optimal solution, while achieving high performance and scalability.

show abstract