Predicting application run times using historical information

Smith, Warren; Foster, Ian; Taylor, Valerie

doi:10.1007/bfb0053984

Cited by 212 publications

(113 citation statements)

References 8 publications

Supporting

Mentioning

110

Contrasting

Unclassified

Order By: Relevance

“…Workload traces have revealed that users tend to frequently resubmit similar or even identical jobs [9,10], a phenomenon that automated runtime predictors have leveraged in the past [11,22]. A symbiotic scheduler may utilize these same techniques to identify applications and associated resource bottlenecks.…”

Section: Identifying Symbiosismentioning

confidence: 99%

Symbiotic Space-Sharing on SDSC’s DataStar System

Weinberg

Snavely

2007

Job Scheduling Strategies for Parallel Processing

View full text Add to dashboard Cite

Abstract.Using a large HPC platform, we investigate the effectiveness of "symbiotic space-sharing", a technique that improves system throughput by executing parallel applications in combinations and configurations that alleviate pressure on shared resources. We demonstrate that relevant benchmarks commonly suffer a 10-60% penalty in runtime efficiency due to memory resource bottlenecks and up to several orders of magnitude for I/O. We show that this penalty can be often mitigated, and sometimes virtually eliminated, by symbiotic space-sharing techniques and deploy a prototype scheduler that leverages these findings to improve system throughput by 20%.

show abstract

Section: Identifying Symbiosismentioning

confidence: 99%

Symbiotic Space-Sharing on SDSC’s DataStar System

Weinberg

Snavely

2007

Job Scheduling Strategies for Parallel Processing

View full text Add to dashboard Cite

show abstract

“…In the works by Smith et al [9] [10], runtime predictions are derived using similar runs in the history, and these estimates are further used to simulate the scheduling algorithms like FCFS, LWF (Least Work First) and Backfilling [11] to obtain the queue wait times predictions. Another work by Li et al [12] tries to improve the runtime prediction and simulate the batch system for the Maui scheduler [4].…”

Section: Related Workmentioning

confidence: 99%

“…Moreover, their efforts use runtime estimates for the prediction of queue wait times. The runtime estimates reported in these efforts [9] [10] have high prediction errors from 33% to 74% in many cases, and hence using these estimates to predict queue wait times will lead to large errors in wait time predictions.…”

Section: Related Workmentioning

confidence: 99%

Identifying Quick Starters: Towards an Integrated Framework for Efficient Predictions of Queue Waiting Times of Batch Parallel Jobs

Kumar

Vadhiyar

2013

Job Scheduling Strategies for Parallel Processing

View full text Add to dashboard Cite

Abstract. Production parallel systems are space-shared and hence employ batch queues in which the jobs submitted to the systems are made to wait before execution. Thus, jobs submitted to parallel batch systems incur queue waiting times in addition to the execution times. Prediction of these queue waiting times is important to provide overall estimates to the users and can also help metaschedulers make scheduling decisions. Analyses of the job traces of supercomputers reveal that about 56 to 99% of the jobs incur queue waiting times of less than an hour. Hence, identifying these quick starters or jobs with short queue waiting times is essential for overall improvement on queue waiting time predictions. Existing strategies provide high overestimates of upper bounds of queue waiting times rendering the bounds less useful for jobs with short queue waiting times. In this work, we have developed an integrated framework that uses the job characteristics, and states of the queue and processor occupancy to identify and predict quick starters, and use the existing strategies to predict jobs with long queue waiting times. Our experiments with different production supercomputer job traces show that our prediction strategies can lead to correct identification of up to 20 times more quick starters and provide tighter bounds for these jobs, and thus result in up to 64% higher overall prediction accuracy than existing methods.

show abstract

“…Since demand and supply of Grid services fluctuates in the course of time, information about current and future resource utilization and prices are often not known a-priori to the participants. In this case consumer and provider agents try to maximize their utilities by generating bids based on their valuations and historic experiences [6]. This results in strategic behavior on provider and consumer side.…”

Section: Automated Resource Allocationmentioning

confidence: 99%

Q-Strategy: A Bidding Strategy for Market-Based Allocation of Grid Services

Borissov

Wirström

2008

On the Move to Meaningful Internet Systems: OTM 2008

View full text Add to dashboard Cite

Abstract. The application of autonomous agents by the provisioning and usage of computational services is an attractive research field. Various methods and technologies in the area of artificial intelligence, statistics and economics are playing together to achieve i) autonomic service provisioning and usage of Grid services, to invent ii) competitive bidding strategies for widely used market mechanisms and to iii) incentivize consumers and providers to use such market-based systems. The contributions of the paper are threefold. First, we present a bidding agent framework for implementing artificial bidding agents, supporting consumers and providers in technical and economic preference elicitation as well as automated bid generation by the requesting and provisioning of Grid services. Secondly, we introduce a novel consumer-side bidding strategy, which enables a goal-oriented and strategic behavior by the generation and submission of consumer service requests and selection of provider offers. Thirdly, we evaluate and compare the Q-strategy, implemented within the presented framework, against the Truth-Telling bidding strategy in three mechanisms -a centralized CDA, a decentralized on-line machine scheduling and a FIFO-scheduling mechanisms.

show abstract

Predicting application run times using historical information

Abstract: The Soudan 2 detector is used to search for evidence of nucleon decay. Particular emphasis is put on searches for modes with multiple-charged particles in the final st ate, and for modes suggested by super-symmetric theories.Mode

Cited by 212 publications

References 8 publications

Symbiotic Space-Sharing on SDSC’s DataStar System

Symbiotic Space-Sharing on SDSC’s DataStar System

Identifying Quick Starters: Towards an Integrated Framework for Efficient Predictions of Queue Waiting Times of Batch Parallel Jobs

Q-Strategy: A Bidding Strategy for Market-Based Allocation of Grid Services

Contact Info

Product

Resources

About