Continuous learning of HPC infrastructure models using big data analytics and in-memory processing tools

Abstract

Exascale computing represents the next leap in the HPC race. Reaching this level of performance is subject to several engineering challenges such as energy consumption, equipmentcooling, reliability and massive parallelism. Model-based optimization is an essential tool in the design process and control of energy efficient, reliable and thermally constrained systems. However, in the Exascale domain, model learning techniques tailored to the specific supercomputer require real measurements and must therefore handle and analyze a massive amount of data coming from the HPC monitoring infrastructure. This becomes rapidly a "big data" scale problem. The common approach where measurements are first stored in large databases and then processed is no more affordable due to the increasingly storage costs and lack of real-time support. Nowadays instead, cloudbased machine learning techniques aim to build on-line models using real-time approaches such as "stream processing" and "inmemory" computing, that avoid storage costs and enable fastdata processing. Moreover, the fast delivery and adaptation of the models to the quick data variations, make the decision stage of the optimization loop more effective and reliable. In this paper we leverage scalable, lightweight and flexible IoT technologies, such as the MQTT protocol, to build a highly scalable HPC monitoring infrastructure able to handle the massive sensor data produced by next-gen HPC components. We then show how state-of-the art tools for big data computing and analysis, such as Apache Spark, can be used to manage the huge amount of data delivered by the monitoring layer and to build adaptive models in real-time using on-line machine learning techniques.

Online Anomaly Detection in HPC Systems

“…These agents monitor the power consumption of each node at the plug as well as performance and utilization metrics, using both software commands and hardware sensors. The measured values are sent to a data management backbone (namely ExaMon [BBCB17]), through a communication layer based on the opensource MQTT (MQ Telemetry Transport) protocol [Sta14], which is designed for low bandwidth, high latency networks and minimal resource demands.…”

“…The integrated monitoring infrastructure periodically reads a set of metrics and collects them into a single gathering point. The authors of [BBCB17] show that these sensors can easily reach 1.5KSa/s per compute node, and propose Examon, a scalable infrastructure based on local monitoring agents pushing data through the MQTT protocol. Clearly, local software-based monitoring agents compete for the same computational resources of users' applications.…”

Anomaly Detection Using Autoencoders in High Performance Computing Systems

“…The data collection infrastructure deployed in D.A.V.I.D.E. is called Examon and has been presented in previous works [BBCea17,BBLea18]. Examon is a fine-grained, lightweight and scalable monitoring infrastructure for Exascale supercomputers.…”

The ANTAREX domain specific language for high performance computing

“…These elements include the node and rack cooling components as well as environmental parameters such as the room and ambient temperature. In ANTAREX, we developed ExaMon [26] (Exascale Monitoring) to virtualise the performance and power monitoring access in a distributed environment. ExaMon decouples the sensor readings from the sensor value usage.…”