Job management requirements for nas parallel systems and clusters

Saphir, William; Tanner, Leigh; Traversat, Bernard

doi:10.1007/3-540-60153-8_37

Cited by 10 publications

(7 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The issues of timesharing the NAS workload on parallel systems was discussed in detail several years ago [10]. The following excerpt is relevant here:…”

Section: Supercomputer Resource Sharingmentioning

confidence: 99%

See 1 more Smart Citation

Scheduling for Parallel Supercomputing: A Historical Perspective of Achievable Utilization

Jones

Nitzberg

1999

Job Scheduling Strategies for Parallel Processing

View full text Add to dashboard Cite

Abstract.The NAS facility has operated parallel supercomputers for the past 11 years, including the Intel iPSC/860, Intel Paragon, Thinking Machines CM-5, IBM SP-2, and Cray Origin 2000. Across this wide variety of machine architectures, across a span of 10 years, across a large number of different users, and through thousands of minor configuration and policy changes, the utilization of these machines shows three general trends: (1) scheduling using a naive FCFS first-fit policy results in 40-60% utilization, (2) switching to the more sophisticated dynamic backfilling scheduling algorithm improves utilization by about 15 percentage points (yielding about 70% utilization), and (3) reducing the maximum allowable job size further increases utilization. Most surprising is the consistency of these trends. Over the lifetime of the NAS parallel systems, we made hundreds, perhaps thousands, of small changes to hardware, software, and policy, yet utilization was affected little. In particular, these results show that the goal of achieving near 100% utilization while supporting a real parallel supercomputing workload is unrealistic.

show abstract

“…The issues of timesharing the NAS workload on parallel systems was discussed in detail several years ago [10]. The following excerpt is relevant here:…”

Section: Supercomputer Resource Sharingmentioning

confidence: 99%

“…(For a discussion of batch job scheduling requirements for large parallel supercomputers, like those at NAS, see [10].) During the analysis, several trends begin to emerge.…”

Section: Analysis Of Nas Parallel Supercomputer Utilizationmentioning

confidence: 99%

Scheduling for Parallel Supercomputing: A Historical Perspective of Achievable Utilization

Jones

Nitzberg

1999

Job Scheduling Strategies for Parallel Processing

View full text Add to dashboard Cite

show abstract

“…In the SAN architecture, the unrelated backup tasks could be executed in a FIFO style [Willim et al 1995]. This algorithm is easy to realize, and executes rapidly.…”

Section: Methodsmentioning

confidence: 98%

Research and implementation of management software for SAN based on clustering technology

Deng

Zheng

Wang

et al. 2004

Proceedings of the 2004 ACM SIGGRAPH International Conference on Virtual Reality Continuum and Its Applications in Industry - V

View full text Add to dashboard Cite

We present a new SAN architecture by introducing the advantage of clustering technology into traditional SAN system, and discuss the implement of this system. Furthermore, a mathematical model and the corresponding algorithm are developed for representing related backup task. A scheduling algorithm is also developed on the basis of the quasi ordering for related backup task in timing submittal. Experiment results show that this algorithm can improve the performance of scheduling and executing related backup tasks.

show abstract

“…Job Types This architecture has stood the test of time; it is not much different from those first developed for parallel computing schedulers , Saphir et al (1995)].…”

Section: Job Arraysmentioning

confidence: 99%

Scalable system scheduling for HPC and big data

Reuther

Byun

Arcand

et al. 2018

Journal of Parallel and Distributed Computing

View full text Add to dashboard Cite

In the rapidly expanding field of parallel processing, job schedulers are the "operating systems" of modern big data architectures and supercomputing systems. Job schedulers allocate computing resources and control the execution of processes on those resources. Historically, job schedulers were the domain of supercomputers, and job schedulers were designed to run massive, long-running computations over days and weeks. More recently, big data workloads have created a need for a new class of computations consisting of many short computations taking seconds or minutes that process enormous quantities of data. For both supercomputers and big data systems, the efficiency of the job scheduler represents a fundamental limit on the efficiency of the system. Detailed measurement and modeling of the performance of schedulers are critical for maximizing the performance of a large-scale computing system. This paper presents a detailed feature analysis of 15 supercomputing and big data schedulers. For big data workloads, the scheduler latency is the most important performance characteristic of the scheduler. A theoretical model of the latency of these schedulers is developed and used to design experiments targeted at measuring scheduler latency. Detailed benchmarking of four of the most popular schedulers (Slurm, Son of Grid Engine, Mesos, and Hadoop YARN) are conducted. The theoretical model is compared with data and demonstrates that scheduler performance can be characterized by two key parameters: the marginal latency of the scheduler t s and a nonlinear exponent α s . For all four schedulers, the utilization of the computing system decreases to <10% for computations lasting only a few seconds. Multi-level schedulers (such as LLMapReduce) that transparently aggregate short computations can improve utilization for these short computations to >90% for all four of the schedulers that were tested.

show abstract

Job management requirements for nas parallel systems and clusters

Cited by 10 publications

References 1 publication

Scheduling for Parallel Supercomputing: A Historical Perspective of Achievable Utilization

Scheduling for Parallel Supercomputing: A Historical Perspective of Achievable Utilization

Research and implementation of management software for SAN based on clustering technology

Scalable system scheduling for HPC and big data

Contact Info

Product

Resources

About