Autonomous Detection of Thermal Anomalies in Data Centers

Abstract

In recent years, there has been a significant growth in number, size and power densities of data centers. A significant part of data center power consumption is attributed to the cooling infrastructure, consisting of computer air conditioning units (CRACs), chillers and cooling towers. For energy efficient operation and management of the cooling resources, data centers are beginning to be extensively instrumented with temperature sensors. While this allows cooling actuators, such as CRAC set point temperature, to be dynamically controlled and data centers operated at higher temperatures to save energy, it also increases chances of thermal anomalies. Furthermore, considering that large data centers can contain thousands to tens of thousands of such sensors, it is virtually impossible to manually inspect and analyze the large volumes of dynamic data generated by these sensors, thus necessitating autonomous mechanisms for thermal anomaly detection. Also, in addition to threshold-based detection methods, other mechanisms of anomaly detection are also necessary. In this paper, we describe the commonly occurring thermal anomalies in a data center. Furthermore, we describe — with examples from a production data center — techniques to autonomously detect these anomalies. In particular, we show the usefulness of a principal component analysis (PCA) based methodology to a large temperature sensor network. Specifically, we examine thermal anomalies such as those related to misconfiguration of equipment, blocked vent tiles, faulty sensor and CRAC related anomalies. Furthermore, several of these anomalies normally go undetected since no temperature thresholds are violated. We present examples of the thermal anomalies and their detection from a real data center.

Thermal Management in Large Data Centres: Security Threats and Mitigation

“…The study of thermal anomaly detection specifically designed for a data centre environment has only started to gain attention during the last decade, where there has been a significant development of data warehouses around the globe. Studies published by Marwah et al [25,26,27] in 2009-10 were the first to ad-dress the issue of autonomous anomaly detection of temperature values in data centres and apply machine learning on sensor data sets. In their first paper on the topic, [27] underlined the importance for an autonomous system that is able to catch temperature values beyond a certain threshold.…”

“…Studies published by Marwah et al [25,26,27] in 2009-10 were the first to ad-dress the issue of autonomous anomaly detection of temperature values in data centres and apply machine learning on sensor data sets. In their first paper on the topic, [27] underlined the importance for an autonomous system that is able to catch temperature values beyond a certain threshold. They also listed common technical reasons, as well as their symptoms, that lead to anomalies.…”

Thermal anomaly detection in datacenters

“…Then, it detects thermal anomalies by evaluating deviations of the estimated temperatures (from the thermal map) from actual temperatures. 1,4,5 Machine-learning-based approach is used to learn thermal behaviors in datacenters by training and compare the results with the actual temperatures to detect the anomalies.…”

“…Such events will lead to unexpected anomalies like thermal hotspots and fugues. 1 Thermal anomalies can be small (spanning a few servers or racks) or large (spanning many servers or racks) in scale, causing severe performance degradation of server hardware. Hence, these thermal anomalies need to be detected, classified (with respect to the anomalous events that caused them), and localized for timely remedial action.…”