An Infrastructure for Efficient Parallel Job Execution in Terascale Computing Environments

Moreira, José E.; Chan, Waiman; Fong, Liana; Franke, Hubertus; Jette, Morris A.

doi:10.1109/sc.1998.10026

Cited by 22 publications

(23 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, we have shown in previous work [7,15] that gangscheduling is very effective in improving the performance of FCFS policies. This is in agreement with the results P 0 P 1 P 2 P 3 P 4 P 5 P 6 P 7 in [4,17].…”

Section: Considerations In Building a Scheduling Matrixmentioning

confidence: 84%

An Integrated Approach to Parallel Scheduling Using Gang-Scheduling, Backfilling, and Migration

Zhang

Franke

Moreira

et al. 2001

Job Scheduling Strategies for Parallel Processing

Self Cite

View full text Add to dashboard Cite

Effective scheduling strategies to improve response times, throughput, and utilization are an important consideration in large supercomputing environments. Such machines have traditionally used space-sharing strategies to accommodate multiple jobs at the same time. This approach, however, can result in low system utilization and large job wait times. This paper discusses three techniques that can be used beyond simple space-sharing to greatly improve the performance figures of large parallel systems. The first technique we analyze is backfilling, the second is gang-scheduling, and the third is migration. The main contribution of this paper is an evaluation of the benefits from combining the above techniques. We demonstrate that, under certain conditions, a strategy that combines backfilling, gang-scheduling, and migration is always better than the individual strategies for all quality of service parameters that we consider.

show abstract

Section: Considerations In Building a Scheduling Matrixmentioning

confidence: 84%

An Integrated Approach to Parallel Scheduling Using Gang-Scheduling, Backfilling, and Migration

Zhang

Franke

Moreira

et al. 2001

Job Scheduling Strategies for Parallel Processing

Self Cite

View full text Add to dashboard Cite

show abstract

“…Gang scheduling has inspired a considerable amount of academic research for a number of years [43][44][45][46][47]. While there have been several promising implementations of the concept [48][49][50], gang scheduling remains very much a curiosity in HPC circles and is not generally found in production environments (a notable exception being the CM5 [35], no longer in use). This is largely because the promised benefits have been difficult to realize [46].…”

Section: Gang Schedulingmentioning

confidence: 99%

Dynamic Fractional Resource Scheduling for cluster platforms

Stillwell

2010

2010 IEEE International Symposium on Parallel &Amp; Distributed Processing, Workshops and PHD Forum (IPDPSW)

View full text Add to dashboard Cite

Experiments presented in this paper were carried out using the Grid'5000 experimental testbed, being developed under the INRIA ALADDIN development action with support from CNRS, RENATER and several Universities as well as other funding bodies (see https://www.grid5000.fr).The experiments using our prototype system were carried out on machines owned by the Deparment of Computer Science and Engineering at the University of California, San Diego with the permission of Geoffrey Voelker. System administration and much assistance were provided by Gjergji Zyba. iv ABSTRACTThis research focuses on the problem of job scheduling on homogeneous computational clusters. Clusters are widely used today for a variety of purposes, including high-performance scientific computing and Internet service hosting. While clusters may have impressive aggregate performance metrics, they are really only collections of fairly modest machines, which makes scheduling jobs for the best performance a non-trivial problem. Most clusters also need to be shared among users to amortize their start-up and maintenance costs, and ensuring that these users are treated fairly further adds to the difficulty. Existing approaches to scheduling attempt to address both of these issues, but have several limitations.We propose a novel approach, called Dynamic Fractional Resource Scheduling (DFRS), to sharing homogeneous cluster computing platforms among competing jobs.The key features of DFRS are that it leverages existing virtual machine technology in order to share resources more efficiently and it defines and optimizes a user-centric metric that captures notions of both performance and fairness. In this dissertation we explain the principles behind DFRS and its advantages over the current state of the art, develop a theoretical model of resource sharing, design heuristics to optimize the proposed metric within the given framework, implement and run simulations comparing DFRS to traditional approaches using popular and accepted performance metrics, and finally develop and test a prototype implementation based on existing technologies. Our results show that it is possible to develop heuristic algorithms that give results reasonably close to theoretical bounds for a variety of cases, that resource requirements are well within the capabilities of modern systems, and that for some scenarios DFRS can provide orders-ofmagnitude levels of improvement in performance over current approaches.

show abstract

“…The generic hierarchical communication scheme implemented by LoadLeveler [8] is shown in Figure 2 with an exemplary fan-out number of 3. When a job is scheduled by the Negotiator a job object of assigned machines and job specific information is sent to SchedD.…”

Section: Hierarchical Scheme For Job Dispatchingmentioning

confidence: 99%

Scalability Analysis of Job Scheduling Using Virtual Nodes

Bobroff

Coppinger²,

Fong

et al. 2009

Job Scheduling Strategies for Parallel Processing

View full text Add to dashboard Cite

Abstract. It is important to identify scalability constraints in existing job scheduling software as they are applied to next generation parallel systems. In this paper, we analyze the scalability of job scheduling and job dispatching functions in the IBM LoadLeveler job scheduler. To enable this scalability study, we propose and implement a new virtualization method to deploy different size LoadLeveler clusters with minimal number of physical machines. Our scalability studies with the virtualization show that the LoadLeveler resource manager can comfortably handle over 12,000 compute nodes, the largest scale we have tested so far. However, our study shows that the static resource matching in the scheduling cycle and job object processing during the hierarchical job launching are two impediments for the scalability of LoadLeveler.

show abstract

An Infrastructure for Efficient Parallel Job Execution in Terascale Computing Environments

Cited by 22 publications

References 13 publications

An Integrated Approach to Parallel Scheduling Using Gang-Scheduling, Backfilling, and Migration

An Integrated Approach to Parallel Scheduling Using Gang-Scheduling, Backfilling, and Migration

Dynamic Fractional Resource Scheduling for cluster platforms

Scalability Analysis of Job Scheduling Using Virtual Nodes

Contact Info

Product

Resources

About