Evaluation of synchronization protocols for fine-grain HPC sensor data time-stamping and collection

Abstract

Abstract-Solutions for accurate and fine-grain monitoring are at the basis of the growth of future large-scale green high performance computing (HPC) infrastructures. The capability of these systems to adapt to specific application requirements relies on sensing and correlating several distributed physical parameters with application phases and states. Meeting such requirements allows thus to achieve a better use of the resources, higher throughput and higher energy-efficiency. As the capability of drawing such correlations relies on the synchronization across a network of nodes and measuring devices, the use of synchronization protocols becomes a critical component. Novel low-cost embedded devices start to include hardware support for network synchronization protocols to achieve a high resolution time accuracy. These devices are promising for monitoring physical parameters of HPC infrastructures. In this paper we evaluate how the performance of the two widely used network synchronization protocols, namely the Network Time Protocol and IEEE 1588, scale on a state-of-the-art embedded platform, namely a Beaglebone Black Board.

Online Anomaly Detection in HPC Systems

“…To measure the power consumption with high resolution, the monitoring agents exploit a power sensing module and an embedded monitoring board (Beaglebone Black -BBB [Col13]), one for each node. The BBBs are based on an Arm Cortex-A8, and include a 12-bit ADC which allows sampling rates up to 50k samples per second and synchronization of the measurements within one microsecond, thanks to the hardware support of the Precision Time Protocol (PTP) [LBMB16,LBCB04]. For an out-of-band monitoring of the nodes performance we use the IBM Amester commands, which exploit the IPMI interface [Sla03] to the OpenPOWER POWER8 on-chip controller [Ros15] (OCC), to get OCC sensor readings.…”

Evaluation of NTP/PTP fine-grain synchronization performance in HPC clusters

“…Regarding NTP, we used ntpd to synchronize the system clock of the slaves directly to the system clock of the master node (ntp server), while for PTP we used ptp4l to adjust the PHC of the slaves to the master PHC and exploited phc2sys to constantly update the respective system clocks (on both master and slaves). Basing on the results obtained in our previous work [15] (focus of the paper was to find the optimal polling rate of NTP / PTP daemons to achieve best synchronization performance in a point-to-point link), we tuned their polling rate to the optimal operating point, which corresponds to 0.125 Hz (8 s) for NTP, and 1 Hz and 12 Hz for ptp4l and phc2sys, respectively. Moreover, we set the PTP switch in transparent mode to obtain higher levels of accuracy [8].…”

“…This is not needed to address HPC challenges (such as improving energy efficiency and execution time of applications) and at the same time can be an obstacle for supercomputing centers to carefully install a GPS antenna or an atomic clock. It should be noted that under these assumptions we performed in [15] an early evaluation of NTP / PTP performance, but only on IoT platforms (i.e., BBB) and assuming point-to-point links between them (i.e., no use of switches), thus measurements in a whole HPC cluster -i.e., computing nodes, BBB and switches -are still missing.…”

“…With this test we can finally validate both synchronization protocols to achieve a fine-grain synchronization in SoA HPC clusters. [1,15]. Accordingly to these data rates, both protocols are not critical in terms of scalability.…”

pAElla: Edge AI-Based Real-Time Malware Detection in Data Centers

“…The system is interfaced with the power sensor via a 12-bit 8-channels SAR ADC, and with existing in-band / out-of-band telemetries to collect hardware performance counters (e.g., Amester [39], IPMI [40], and RAPL [17]). Moreover, it includes (i) hardware support for the Precision Time Protocol (PTP), which allows sub-microsecond measurements synchronization [43], [44], (ii) two Programmable Real-Time Units (PRU0 and PRU1), that we exploit for real-time feature extraction on-board, and (iii) an ARM Cortex-A8 processor with NEON technology, useful for DSP processing and edge ML inference (e.g., by leveraging the ARM NN SDK [45], which enables efficient translation of existing NN frameworks -such as TensorFlow -to ARM Cortex-A CPUs); • a scalable distributed data analytics framework, namely ExaMon [15], [16], that we use to collect at a lower rate -from seconds to milliseconds -power and performance activity of the all cluster and thus to carry out cluster-level analytics. To send data from the distributed monitoring agents (i.e., daemons running on the BBBs) to the centralized monitoring unit, we adopted MQTT [46], which is a robust, lightweight and scalable publish-subscribe protocol, already used for large-scale systems both in industry and academia (e.g., Amazon, Facebook, [46], [47]).…”