Qespera: an adaptive framework for prediction of queue waiting times in supercomputer systems

Abstract

Production parallel systems are space-shared, and resource allocation on such systems is usually performed using a batch queue scheduler. Jobs submitted to the batch queue experience a variable delay before the requested resources are granted. Predicting this delay can assist users in planning experiment time-frames and choosing sites with less turnaround times and can also help meta-schedulers make scheduling decisions. In this paper, we present an integrated adaptive framework, Qespera, for prediction of queue waiting times on parallel systems. We propose a novel algorithm based on spatial clustering for predictions using history of job submissions and executions. The framework uses adaptive set of strategies for choosing either distributions or summary of features to represent the system state and to compare with history jobs, varying the weights associated with the features for each job prediction, and selecting a particular algorithm dynamically for performing the prediction depending on the characteristics of the target and history jobs. Our experiments with real workload traces from different production systems demonstrate up to 22% reduction in average absolute error and up to 56% reduction in percentage prediction error over existing techniques. We also report prediction errors of less than 1 h for a majority of the jobs. P. MURALI AND S. VADHIYAR Figure 1. Queue waiting times seen in production HPC systems.On space-sharing systems, it is expected that users request a set of compute nodes for a particular duration of time. With multiple users contending for the compute resources, a batch queue submission incurs time because of waiting in the queue before the resources necessary for its execution are allocated. The queue waiting time ranges from a few seconds to even a few days on most production systems. Figure 1 shows the box plot of queue waiting times observed in the workloads at five production systems featured in parallel workloads archive (PWA) [5]. In each box, the lower and upper edges denote the lower and upper quartile of the set of queue waiting times and the middle line shows the median. The whiskers mark the interquartile deviation (1.5 ) from the median. From the large number of outlier points with high waiting times and the small values of lower quartile and median, we can see that queue waiting times vary widely within and across systems.Prediction of queue waiting times for jobs enables users to estimate job turnaround time, which can be used for planning and task management. Additionally, reliable predictions will allow the users to tune job parameters like requested number of nodes or estimated running time to achieve faster response times. Such predictions can also be efficiently used by a meta-scheduler to make automatic scheduling decisions for selecting the appropriate number of processors and queues for job execution. Usage statistics of predictors like queue bounds estimation from time series (QBETS) [6] that was used to provide bounds on the queue waiting times in TeraGrid [7] a...

Metascheduling of HPC Jobs in Day-Ahead Electricity Markets

“…We evaluated our queue waiting time prediction framework using production supercomputer workload traces with varying site and job characteristics, including two Top500 systems, obtained from Parallel Workloads Archive [21]. The detailed results and analyses are contained in our previous work [17]. In summary, our predictions results in up to 22% reduction in the average absolute error and up to 56% reduction in the percentage prediction errors over existing strategies including QBETS [43] and IBL [44] across workloads.…”

A History-Based Resource Manager for Genome Analysis Workflows Applications on Clusters with Heterogeneous Nodes

“…Different prediction approaches exist in order to estimate the performance of jobs running in clusters. Previous research analyses have focused on predicting execution time of jobs [12,13], waiting time [14,15] and slowdown time of applications spawned when resources are shared [16].…”

Job placement advisor based on turnaround predictions for HPC hybrid clouds

“…In our previous experiments, we generated the parameters by using a GA approach. Murali and Vadhiyar [24] use a computationally cheaper approach: they compute the correlation between each feature and labels and use this value as their estimator's weights. In this section, we compare this strategy with the GA optimization approach, considering both how much faster models can be trained and how accurate such models are.…”

“…More recently, Murali and Vadhiyar [24] proposed a framework called Qespera for prediction of queue waiting times for HPC settings. The proposed framework is based on spatial clustering using history of job submissions and executions.…”