Operational Data Analytics in Practice: Experiences from Design to Deployment in Production HPC Environments

Abstract

As HPC systems grow in complexity, efficient and manageable operation is increasingly critical. Many centers are thus starting to explore the use of Operational Data Analytics (ODA) techniques, which extract knowledge from massive amounts of monitoring data and use it for control and visualization purposes. As ODA is a multifaceted problem, much effort has gone into researching its separate aspects: however, accounts of production ODA experiences are still hard to come across.In this work we aim to bridge the gap between ODA research and production use by presenting our experiences with ODA in production, involving in particular the control of cooling infrastructures and visualization of job data on two HPC systems. We cover the entire development process, from design to deployment, highlighting our insights in an effort to drive the community forward. We rely on open-source tools, which make for a generic ODA framework suitable for most scenarios.

ExaMon-X: A Predictive Maintenance Framework for Automatic Monitoring in Industrial IoT Systems

“…); ii) facility managers, concerned with system-wide issues, such as energy consumption, mortgage costs, and thermal/cooling problems [14]; iii) system accountants, who need to manage and keep track of the different accounts and projects, granting access to users, and providing reports and statistics on the usage of the machines [15]; iv) HPC users, industrial and academic partners who submit jobs, and are typically interested in fast completion times and fair prices [16]. Finally, the diversity of roles implies different use-cases, which often require ad-hoc analytics solutions [17].…”