Job Characteristics on Large-Scale Systems: Long-Term Analysis, Quantification, and Implications

Patel, Tirthak; Liu, Zhengchun; Kettimuthu, Raj; Rich, Paul M.; Allcock, William E.; Tiwari, Devesh

doi:10.1109/sc41405.2020.00088

Cited by 24 publications

(29 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Utilization [18]: This corresponds to the proportion of time spent doing computation that ends up in a success (i.e., the completion of a job before its deadline).…”

Section: Optimization Criteriamentioning

confidence: 99%

“…In addition, we study the following two criteria: Mean response time of successfully completed jobs [7], [18]: This corresponds to the delay between the release of the job and its completion. When deadlines are enforced, this criterion is less important but does provide qualitative information about the various solutions.…”

Section: Optimization Criteriamentioning

confidence: 99%

“…With the advent of parallel computing for many societal advances, traditional computing systems are under a much higher load than was typically the case a few years ago. High-Performance Computing (HPC) systems run with a utilization of almost 100% and a job waiting time of days [18].…”

Section: Introductionmentioning

confidence: 99%

“…In Cloud computing, the typical approach is to have dynamic pricing, with prices increasing as the resources become more scarce [22]. In HPC, high waiting times are usually a deterrent with important negative consequences, such as limiting the number of very large jobs on HPC platforms [18].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Work-in-Progress: Evaluating Task Dropping Strategies for Overloaded Real-Time Systems

Gao

Pallez²,

Robert

et al. 2021

2021 IEEE Real-Time Systems Symposium (RTSS)

View full text Add to dashboard Cite

show abstract

“…Utilization [18]: This corresponds to the proportion of time spent doing computation that ends up in a success (i.e., the completion of a job before its deadline).…”

Section: Optimization Criteriamentioning

confidence: 99%

Section: Optimization Criteriamentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Work-in-Progress: Evaluating Task Dropping Strategies for Overloaded Real-Time Systems

Gao

Pallez²,

Robert

et al. 2021

2021 IEEE Real-Time Systems Symposium (RTSS)

View full text Add to dashboard Cite

show abstract

“…From the relatively few traces that are shared publicly, many are focused on important but specific kinds of workloads, such as tightly-coupled parallel jobs [18], bags of tasks [26], and workflows [65]. Other datasets only include a limited subset of metrics such as power consumption [44], or high-level job information [43]. Only a handful of datasets include low-level server metrics, such as the Microsoft Azure serverless traces [49] or the Solvinity business-critical traces [50].…”

Section: Introductionmentioning

confidence: 99%

A Holistic Analysis of Datacenter Operations: Resource Usage, Energy, and Workload Characterization -- Extended Technical Report

Versluis,

Cetin,

Greeven

et al. 2021

Preprint

View full text Add to dashboard Cite

Improving datacenter operations is vital for the digital society. We posit that doing so requires our community to shift, from operational aspects taken in isolation to holistic analysis of datacenter resources, energy, and workloads. In turn, this shift will require new analysis methods, and open-access, FAIR datasets with fine temporal and spatial granularity. We leverage in this work one of the (rare) public datasets providing fine-grained information on datacenter operations. Using it, we show strong evidence that fine-grained information reveals new operational aspects. We then propose a method for holistic analysis of datacenter operations, providing statistical characterization of node, energy, and workload aspects. We demonstrate the benefits of our holistic analysis method by applying it to the operations of a datacenter infrastructure with over 300 nodes. Our analysis reveals both generic and ML-specific aspects, and further details how the operational behavior of the datacenter changed during the 2020 COVID-19 pandemic. We make over 30 main observations, providing holistic insight into the long-term operation of a large-scale, public scientific infrastructure. We suggest such observations can help immediately with performance engineering tasks such as predicting future datacenter load, and also long-term with the design of datacenter infrastructure.

show abstract

Improving batch schedulers with node stealing for failed jobs

Du,

Marchal,

Pallez

et al. 2024

Concurrency and Computation

View full text Add to dashboard Cite

SummaryAfter a machine failure, batch schedulers typically re‐schedule the job that failed with a high priority. This is fair for the failed job but still requires that job to re‐enter the submission queue and to wait for enough resources to become available. The waiting time can be very long when the job is large and the platform highly loaded, as is the case with typical HPC platforms. We propose another strategy: when a job fails, if no platform node is available, we steal one node from another job , and use it to continue the execution of despite the failure. In this work, we give a detailed assessment of this node stealing strategy using traces from the Mira supercomputer at Argonne National Laboratory. The main conclusion is that node stealing improves the utilization of the platform and dramatically reduces the flow of large jobs, at the price of slightly increasing the flow of small jobs.

show abstract

Job Characteristics on Large-Scale Systems: Long-Term Analysis, Quantification, and Implications

Cited by 24 publications

References 42 publications

Work-in-Progress: Evaluating Task Dropping Strategies for Overloaded Real-Time Systems

Work-in-Progress: Evaluating Task Dropping Strategies for Overloaded Real-Time Systems

A Holistic Analysis of Datacenter Operations: Resource Usage, Energy, and Workload Characterization -- Extended Technical Report

Improving batch schedulers with node stealing for failed jobs

Contact Info

Product

Resources

About