Developing a Toolkit for Task Characteristics Prediction Based on Analysis of Queue’s History of a Supercomputer

Abstract

Empirical studies have repeatedly shown that in High-Performance Computing HPC users’ resource estimations lack accuracy. Therefore, resource underestimation may remove the job at any step ofcomputing and subsequently allocated resources will be wasted. Moreover, resource overestimationalso will waste resources. In this work, to effectively utilize the overall HPC system, we proposed anew approach to predict the required resources such as; number of required CPUs, time slots etc. fornewly submitted job. The study focused on predictive analytics tasks including regression andclassification. A supervised machine learning system, comprising several models, was trained basedon the collection of statistical data including per-job and per-user features collected from the referencequeue systems. Results indicated that adding more features to the dataset improves the predictionaccuracy. The possibility of designing a plugin to apply our machine learning system in practicalapplications was studied. A dynamically connected SLURM SPANK plugin was created that adds the“--predict-time” option and takes control on srun and sbatch commands while they are executed. Itwas found that the plugin enables practical use of our proposed machine learning system.