Execution time prediction for grid infrastructures based on runtime provenance data

Abstract

An accurate performance prediction service can be very useful for resource management and the scheduler service and help them make better resource utilization decisions by providing better execution time estimates. In this paper we present a novel approach of predicting the execution time of computational tasks for Grid infrastructures using machine learning models based on multilayer perceptron combined with a principal feature selection algorithm for selecting the most important runtime features.Our technique uses runtime provenance information as input to multilayer perceptron (MLP) based neural network which trains a model that can predict the execution time of programs with reasonably good accuracy. For the development and training of our machine learning models, we used provenance data collected from three different computational Grids by executing real-world applications. By using our MLP based method we were able to minimise the prediction error to as low as 22% for real-world applications on various Grid infrastructures.

A survey of simulation provenance systems: modeling, capturing, querying, visualization, and advanced utilization

“…It would be interesting to uncover hidden patterns or draw new insights by applying data mining techniques on provenance data. So far, a rich body of existing literature has focused on (i) exploring a workflow pattern that frequently appears [13][14][15], (ii) applying provenance data to scheduling or optimizing simulations and workflows that are in execution [16][17][18], and (iii) estimating when to complete a specified workflow (or simulation) [19,20]. Recently, there has been a growing need to apply a variety of data mining techniques to develop frequent pattern mining and classification to provenance data.…”

“…Malik's group [19] suggested a method of predicting the execution time of a computing job on Grid infrastructures, via machine learning methods. For model training, they utilized provenance data in association with job execution.…”

Selective and Recurring Re-computation of Big Data Analytics Tasks: Insights from a Genomics Case Study

“…The problem of learning cost estimators has been addressed in the recent past, but mainly for specific scenarios that are relevant to data analytics, namely workflow-based programming on clouds and grid [24,25]. But for instance [26] showed that runtime, especially in the case of machine learning algorithms, may depend on features that are specific to the input, and thus not easy to learn.…”

Preserving the Value of Large Scale Data Analytics over Time Through Selective Re-computation

“…Learning cost estimators. This problem has been addressed in the recent past, but mainly for specific scenarios that are relevant to data analytics, namely workflow-based programming on clouds and grid, [17,12]. But for instance [14] showed that runtime, especially in the case of machine learning algorithms, may depend on features that are specific to the input, and thus not easy to learn.…”