Analysis of Job Metadata for Enhanced Wall Time Prediction

Abstract

For efficient utilization of large-scale HPC systems, the task of resource management and job scheduling is of highest priority. Therefore, modern job scheduling systems require information about the estimated total wall time of the jobs already at submission time. Proper wall time estimates are a key for reliable scheduling decisions. Typically, users specify these estimates, already at submission time, based on either previous knowledge or certain limits given by the system. Real-world experience shows that user given estimates are far away from accurate. Hence, an automated system is desirable that creates more precise wall time estimates of submitted jobs. In this paper, we investigate different job metadata and their impact on the wall time prediction. For the job wall time prediction, we used machine learning methods and the workload traces of large HPC systems. In contrast to previous work, we also consider the jobname and in particular the submission directory. Our evaluation shows that we can better predict the accuracy of jobs per user by a factor of seven than most users, without any in-depth analysis of the job.

CLUTCH: A Clustering-Driven Runtime Estimation Scheme for Scientific Simulations

“…With CLUTCH, an opportunity for online simulation platform users to estimate their simulation times a priori will be offered, allowing the users to adjust their simulation schedules accordingly; further, the platform administrators will be able to develop smart schedulers to improve simulation job throughput [2], [3]. Additional technical details are discussed in Section IV.…”