Evalix: Classification and Prediction of Job Resource Consumption on HPC Platforms

Emeras, Joseph; Varrette, Sébastien; Guzek, Mateusz; Bouvry, Pascal

doi:10.1007/978-3-319-61756-5_6

Cited by 8 publications

(11 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To understand the tradeoff among multi-path monolithic, two-level, lock-free and adaptive lock-free schedulers, we build a simulator that is based on a manycore system. This simulator is driven by synthetic workloads and its parameters are drawn from empirical workload distributions [1], [22], [30]. We use this to compare the behavior of multi-path monolithic, two-level, lock-free and adaptive lock-free scheduling models under the same conditions with identical workloads.…”

Section: Simulation Methodologymentioning

confidence: 99%

“…Two-level scheduling models assume that the tasks run on them are largely short-lived and relinquish resource frequently, a valid assumption for data-intensive workloads [21], [22]. However, because HPC workloads are dominated by compute-bound and long-running applications [1], [23], two-level scheduling models are not applicable for HPC nodes.…”

Section: Related Workmentioning

confidence: 99%

“…Our simulators aim to address two main applications: data-intensive applications and traditional MPI applications. There are many approaches to partition workloads among clusters [1], [23]. To simplify and obtain the critical behavior characteristics of different scheduling models, we select a simple two-way split on top of our simulators between I/O-bound jobs that provide end-user operations (e.g., web services) and internal infrastructure services (e.g., BigTable), and compute-bound jobs that perform a computation.…”

Section: Workloadsmentioning

confidence: 99%

“…As discussed [1], [24], we assume that all the tasks of jobs from HPC workloads are mainly focused on CPUintensive, and the task in one-task job are sequential. For data-intensive jobs [40], we assume that a job is made up of one or more tasks (occasionally thousands of tasks), which are mostly CPU-intensive (> 80%) but with fast turnaround, whereas the rest are I/O-intensive (< 20%) and consume the majority of resources.…”

Section: Synthetic Workloadmentioning

confidence: 99%

“…ing the low use of a node's memory and underutilization of a node's CPU resource [1]- [3]. Consequently, various cluster scheduling algorithms (such as POPPA [4], DFRS [5]) and backfilling algorithms [6]- [8] have been developed to allow multiple jobs to coexist on the same node to improve overall turn-around times and fairness, which leads to improving the utilization of HPC systems.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Improving Per-Node Computing Efficiency by an Adaptive Lock-Free Scheduling Model

Zheng

Zhou

et al. 2018

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

Job scheduling on many-core computers with tens or even hundreds of processing cores is one of the key technologies in High Performance Computing (HPC) systems. Despite many scheduling algorithms have been proposed, scheduling remains a challenge for executing highly effective jobs that are assigned in a single computing node with diverse scheduling objectives. On the other hand, the increasing scale and the need for rapid response to changing requirements are hard to meet with existing scheduling models in an HPC node. To address these issues, we propose a novel adaptive scheduling model that is applied to a single node with a many-core processor; this model solves the problems of scheduling efficiency and scalability through an adaptive optimistic control mechanism. This mechanism exposes information such that all the cores are provided with jobs and the tools necessary to take advantage of that information and thus compete for resources in an uncoordinated manner. At the same time, the mechanism is equipped with adaptive control, allowing it to adjust the number of running tools dynamically when frequent conflict happens. We justify this scheduling model and present the simulation results for synthetic and real-world HPC workloads, in which we compare our proposed model with two widely used scheduling models, i.e. multi-path monolithic and two-level scheduling. The proposed approach outperforms the other models in scheduling efficiency and scalability. Our results demonstrate that the adaptive optimistic control affords significant improvements for HPC workloads in the parallelism of the node-level scheduling model and performance.

show abstract

Section: Simulation Methodologymentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Workloadsmentioning

confidence: 99%

Section: Synthetic Workloadmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Improving Per-Node Computing Efficiency by an Adaptive Lock-Free Scheduling Model

Zheng

Zhou

et al. 2018

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

show abstract

Towards bespoke optimizations of energy efficiency in HPC environments

Tracey,

Elisseev,

Smyrnakis

et al. 2023

Applied AI Letters

View full text Add to dashboard Cite

We present bespoke energy efficiency optimizations in high performance computing (HPC) environments using holistic approach to data collection, analysis and proactive management of resources and workloads. Our solution has three major components: (i) platform for collecting, storing and processing data from multiple sources across hardware and software stacks, (ii) collections of regression machine learning (ML) algorithms for making workloads classifications and energy usage predictions, (iii) agent‐based decision‐making framework for delivering control decisions to middle‐ware and infrastructure thus supporting real time or near real energy efficiency optimizations. We will provide some concrete examples of using our proposed approach in HPC environment.

show abstract

Improving Fairness in a Large Scale HTC System Through Workload Analysis and Simulation

Azevedo

Klusáček

Suter

2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Monitoring and analyzing the execution of a workload is at the core of the operation of data centers. It allows operators to verify that the operational objectives are satised or detect and react to any unexpected and unwanted behavior. However, the scale and complexity of large workloads composed of millions of jobs executed each month on several thousands of cores, often limit the depth of such an analysis. This may lead to overlook some phenomena that, while not harmful at a global scale, can be detrimental to a specic class of users. In this paper, we illustrate such a situation by analyzing a large High Throughput Computing (HTC) workload trace coming from one of the largest academic computing centers in France. The Fair-Share algorithm at the core of the batch scheduler ensures that all user groups are fairly provided with an amount of computing resources commensurate to their expressed needs. However, a deeper analysis of the produced schedule, especially of the job waiting times, shows a certain degree of unfairness between user groups. We identify the conguration of the quotas and scheduling queues as the main root causes of this unfairness. We thus propose a drastic reconguration of the system that aims at being more suited to the characteristics of the workload and at better balancing the waiting time among user groups. We evaluate the impact of this reconguration through detailed simulations. The obtained results show that it still satises the main operational objectives while signicantly improving the quality of service experienced by formerly unfavored users.

show abstract

Evalix: Classification and Prediction of Job Resource Consumption on HPC Platforms

Cited by 8 publications

References 21 publications

Improving Per-Node Computing Efficiency by an Adaptive Lock-Free Scheduling Model

Improving Per-Node Computing Efficiency by an Adaptive Lock-Free Scheduling Model

Towards bespoke optimizations of energy efficiency in HPC environments

Improving Fairness in a Large Scale HTC System Through Workload Analysis and Simulation

Contact Info

Product

Resources

About