Using Probabilistic Characterization to Reduce Runtime Faults in HPC Systems

Abstract

The current trend in high performance computing is to aggregate ever larger numbers of processing and interconnection elements in order to achieve desired levels of computational power, This, however, also comes with a decrease in the Mean Time To Interrupt because the elements comprising these systems are not becoming significantly more robust. There is substantial evidence that the Mean Time To Interrupt vs. number of processor elements involved is quite similar over a large number of platforms. In this paper we present a system that uses hardware level monitoring coupled with statistical analysis and modeling to select processing system elements based on where they lie in the statistical distribution of similar elements. These characterizations can be used by the scheduler/resource manager to deliver a close to optimal set of processing elements given the available pool and the reliability requirements of the application.

Resource monitoring and management with OVIS to enable HPC in cloud computing environments

“…Using these techniques saves time relative to the case of over-aggressive checkpointing and still ensures progress. Work has also been done in the area of predictive analysis by Stearley and Oliner [9] on the Sisyphus project, which seeks to discover correlations of log-file events with both software-and hardware-related failures, and by the OVIS project [3], which looks for correlations of multi-variate hardware state behaviors with failures. The motivation for the latter work was that targeted checkpointing, based upon failure prediction, could dramatically increase the scalability of HPC applications and platforms, since checkpointing all state for the application would no longer be necessary and additionally state would only have to be saved for affected processes when they were deemed destined to fail.…”

Developing a power measurement framework for cyber defense

“…Board, core or disk temperatures correlate to failures in some studies but not others [7,8]. Some systems collect hundreds of variables per node, analyze on the fly, and save only analysis results, thus preventing comprehensive forensics or reuse of their data [9,10]. Hsu and Poole in [11] detail state of the art in power measurement and classify hardware monitoring methods from node component level to facility.…”

AccaSim: a customizable workload management simulator for job dispatching research in HPC systems

“…Additional data. It has been shown in the last decade that system performance can be enhanced greatly if the dispatchers are aware of additional information regarding the current system status, such as energy and power consumption of the resources [37,2,5,6], resource failures [22,7], and the heating/cooling conditions [35,3]. The additional data component of AccaSim provides an interface to integrate such extra data to the system which can then be utilized to develop and experiment with advanced dispatchers which are for instance energy and power-aware, fault-resilient and thermal-aware.…”