Job Scheduling in High Performance Computing

Abstract

The ever-growing processing power of supercomputers in recent decades enables us to explore increasing complex scientific problems. Effective scheduling these jobs is crucial for individual job performance and system efficiency. The traditional job schedulers in high performance computing (HPC) are simple and concentrate on improving CPU utilization. The emergence of new hardware resources and novel hardware structure impose severe challenges on traditional schedulers. The increasing diverse workloads, including compute-intensive and data-intensive applications, require more efficient schedulers. Even worse, the above two factors interplay with each other, which makes scheduling problem even more challenging. In recent years, many research has discussed new scheduling methods to combat the problems brought by rapid system changes. In this research study, we have investigated challenges faced by HPC scheduling and state-of-art scheduling methods to overcome these challenges. Furthermore, we propose an intelligent scheduling framework to alleviate the problems encountered in modern job scheduling.

Performance-driven scheduling for malleable workloads

“…For moldable jobs, the scheduler makes the decision at the beginning of execution. Malleable and evolving jobs can dynamically shrink or grow the resources on which they execute during runtime [36]. By shrink, a malleable job releases part of the assigned resources, allowing the start of some new jobs waiting in the queue or reconfiguring other malleable jobs that can benefit from these released resources.…”