Effective Prediction of Job Processing Times in a Large-Scale Grid Environment

Dobber, Menno; Mei, Rob van der; Koole, Ger

doi:10.1109/hpdc.2006.1652183

Cited by 15 publications

(12 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Existing works [17,7,22] and [19] have focused on the problem of execution time prediction of tasks. Some of these existing works have also utilized the historical data to create statistical models and machine learning based models to determine execution times with a high accuracy.…”

Section: Introductionmentioning

confidence: 99%

Execution time prediction for grid infrastructures based on runtime provenance data

Malik

Fahringer

Prodan

2013

Proceedings of the 8th Workshop on Workflows in Support of Large-Scale Science

View full text Add to dashboard Cite

An accurate performance prediction service can be very useful for resource management and the scheduler service and help them make better resource utilization decisions by providing better execution time estimates. In this paper we present a novel approach of predicting the execution time of computational tasks for Grid infrastructures using machine learning models based on multilayer perceptron combined with a principal feature selection algorithm for selecting the most important runtime features.Our technique uses runtime provenance information as input to multilayer perceptron (MLP) based neural network which trains a model that can predict the execution time of programs with reasonably good accuracy. For the development and training of our machine learning models, we used provenance data collected from three different computational Grids by executing real-world applications. By using our MLP based method we were able to minimise the prediction error to as low as 22% for real-world applications on various Grid infrastructures.

show abstract

Section: Introductionmentioning

confidence: 99%

Execution time prediction for grid infrastructures based on runtime provenance data

Malik

Fahringer

Prodan

2013

Proceedings of the 8th Workshop on Workflows in Support of Large-Scale Science

View full text Add to dashboard Cite

show abstract

“…To estimate these costs we analysed the provenance traces of our case. Other work [6] [7] [8] describe how to predict those information using execution logs or workflow input data.…”

Section: Introductionmentioning

confidence: 99%

A Scalable Method for Partitioning Workflows with Security Requirements over Federated Clouds

Wen

Cała

Watson

2014

2014 IEEE 6th International Conference on Cloud Computing Technology and Science

View full text Add to dashboard Cite

Abstract-The significant increase in the use of cloud computing, has led to an interest in partitioning applications over a set of public and private clouds in order to meet a range of non-functional requirements including performance (for example where private cloud resources alone are insufficient), dependability (e.g. to allow the application to continue to operate even if one cloud fails) and security (for example to ensure that sensitive data is restricted to sufficiently secure clouds and networks). This paper describes a novel deployment planning algorithm to partition complex workflow-based applications over federated clouds, while meeting security requirements. The security issues are based on our previous work which extends the Bell-LaPadula model to encompass cloud computing. Selecting the cheapest option for partitioning a workflow over a set of resources has been shown to be an NP-hard problem, which can take impractically long for partitioning large workflows over multiple clouds. We therefore introduce a novel adaptive partitioning algorithm to handle these large workflow applications, which significantly reduces the time required to choose a sufficientlygood partitioning option. This is based on generating an initial partitioning, and then adapting it to see if a better solution can be found by bringing together on the same node services with significant communication costs. The algorithm has been implemented and evaluated by using both randomly generated and real world scientific workflows. The experiment results show that our algorithm is thousands times quicker than the exhaustive algorithm presented in our previous work. Yet, on average it generates only 25% more costly solutions. We also compared this algorithm with two other methods commonly used to partition workflows over a set of clouds.

show abstract

“…Several works [10], [11], and [12] investigate task runtime prediction from their past performance, on the same machines. This approach derives from their context of application: a shared grid, where tasks controlled elsewhere are running on the same grid.…”

Section: A Distributed Computingmentioning

confidence: 99%

A Review on Task Performance Prediction in Multi-core Based Systems

Pinel

Pecero

Bouvry

et al. 2011

2011 IEEE 11th International Conference on Computer and Information Technology

View full text Add to dashboard Cite

Abstract-Operators of data centers are faced with the challenging goal of hosting applications that meet agreed service levels, at minimal operating costs. A significant part of these costs is energy related. Successfully reaching this goal requires optimal task-to-machine assignments. This activity relies on accurate energy and performance prediction. Widespread use of multi-core, multi-processor machines complicate past prediction methods. Therefore, this paper suggests to revisit task profiling, a method based on observations of actual task execution. As a first step in this direction, this paper reviews methods for task profiling, which account for the contention present in multi-core processors.

show abstract

Effective Prediction of Job Processing Times in a Large-Scale Grid Environment

Cited by 15 publications

References 8 publications

Execution time prediction for grid infrastructures based on runtime provenance data

Execution time prediction for grid infrastructures based on runtime provenance data

A Scalable Method for Partitioning Workflows with Security Requirements over Federated Clouds

A Review on Task Performance Prediction in Multi-core Based Systems

Contact Info

Product

Resources

About