Machine Learning Models for GPU Error Prediction in a Large Scale HPC System

Nie, Bin; Xue, Ji; Gupta, Saurabh; Patel, Tirthak; Engelmann, Christian; Smirni, Evgenia; Tiwari, Devesh

doi:10.1109/dsn.2018.00022

Cited by 61 publications

(25 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although some of these studies utilize workload inherent features, such as the memory reuse time, they are orthogonal to our work. Predictive maintenance and statistical prediction of errors: Considerable research has been done on statistical pre-diction of different types of hardware faults, including DRAM errors, in supercomputers [4], [18], [35], [38], [44], [58], [66], [89]. The majority of these studies proposed different techniques, based either on rules [38] or Machine Learning [66], for prediction of failures that may happen in various hardware components using history of errors.…”

Section: Related Workmentioning

confidence: 99%

Workload-Aware DRAM Error Prediction using Machine Learning

Mukhanov

Tovletoglou

Vandierendonck

et al. 2019

2019 IEEE International Symposium on Workload Characterization (IISWC)

View full text Add to dashboard Cite

The aggressive scaling of technology may have helped to meet the growing demand for higher memory capacity and density, but has also made DRAM cells more prone to errors. Such a reality triggered a lot of interest in modeling DRAM behavior for either predicting the errors in advance or for adjusting DRAM circuit parameters to achieve a better tradeoff between energy efficiency and reliability. Existing modeling efforts may have studied the impact of few operating parameters and temperature on DRAM reliability using custom FPGAs setups, however they neglected the combined effect of workloadspecific features that can be systematically investigated only on a real system.In this paper, we present the results of our study on workloaddependent DRAM error behavior within a real server considering various operating parameters, such as the refresh rate, voltage and temperature. We show that the rate of single-and multi-bit errors may vary across workloads by 8x, indicating that program inherent features can affect DRAM reliability significantly. Based on this observation, we extract 249 features, such as the memory access rate, the rate of cache misses, the memory reuse time and data entropy, from various compute-intensive, caching and analytics benchmarks. We apply several supervised learning methods to construct the DRAM error behavior model for 72 servergrade DRAM chips using the memory operating parameters and extracted program inherent features. Our results show that, with an appropriate choice of program features and supervised learning method, the rate of single-and multi-bit errors can be predicted for a specific DRAM module with an average error of less than 10.5 %, as opposed to the 2.9x estimation error obtained for a conventional workload-unaware error model. Our model enables designers to predict DRAM errors in advance for less than a second and study the impact of any workload and applied software optimizations on DRAM reliability.

show abstract

Section: Related Workmentioning

confidence: 99%

Workload-Aware DRAM Error Prediction using Machine Learning

Mukhanov

Tovletoglou

Vandierendonck

et al. 2019

2019 IEEE International Symposium on Workload Characterization (IISWC)

View full text Add to dashboard Cite

show abstract

“…A previous study that analyzes GPU errors on the same system [48] reports that 98% of the detected errors come from the L2 cache. The findings of Nie et al [47], therefore, could not be directly applied to device DRAM error prediction.…”

Section: Gpu Memory Errorsmentioning

confidence: 99%

“…A few recent studies have analysed GPU errors in the field [46], [47]. Nie et al [47] analyze the GPU errors on the Titan supercomputer, which comprises 18,688 K20X GPUs.…”

Section: Gpu Memory Errorsmentioning

confidence: 99%

“…A few recent studies have analysed GPU errors in the field [46], [47]. Nie et al [47] analyze the GPU errors on the Titan supercomputer, which comprises 18,688 K20X GPUs. The authors study the system conditions and workload characteristics that trigger GPU errors, and they propose and evaluate several machine learning-based models: logistic regression, gradient boosting decision tree, support vector machine, and neural network.…”

Section: Gpu Memory Errorsmentioning

confidence: 99%

See 1 more Smart Citation

Cost-Aware Prediction of Uncorrected DRAM Errors in the Field

Boixaderas

Zivanovic

Moré

et al. 2020

SC20: International Conference for High Performance Computing, Networking, Storage and Analysis

View full text Add to dashboard Cite

This paper presents and evaluates a method to predict DRAM uncorrected errors, a leading cause of hardware failures in large-scale HPC clusters. The method uses a random forest classifier, which was trained and evaluated using error logs from two years of production of the MareNostrum 3 supercomputer. By enabling the system to take measures to mitigate node failures, our method reduces lost compute time by up to 57%, a net saving of 21,000 node-hours per year. We release all source code as open source.We also discuss and clarify aspects of methodology that are essential for a DRAM prediction method to be useful in practice. We explain why standard evaluation metrics, such as precision and recall, are insufficient, and base the evaluation on a cost-benefit analysis. This methodology can help ensure that any DRAM error predictor is clear from training bias and has a clear cost-benefit calculation.

show abstract

“…Nie et al [18] continue this work with the analysis of the GPU-error related data on the same system from February 2015 to June 2015. The follow-up study from the same team [19] proposes and evaluates several machine learning-based models for the GPU error prediction. The studies reveal interesting insights about the temporal and spatial distribution of GPU errors, their correlation with temperature, GPU power consumption and workload characteristics.…”

Section: Related Workmentioning

confidence: 99%

DRAM errors in the field

Zivanovic

Dokht

Moré

et al. 2019

Proceedings of the International Symposium on Memory Systems

View full text Add to dashboard Cite

This paper summarizes our two-year study of corrected and uncorrected errors on the MareNostrum 3 supercomputer, covering 2000 billion MB-hours of DRAM in the field. The study analyzes 4.5 million corrected and 71 uncorrected DRAM errors and it compares the reliability of DIMMs from all three major memory manufacturers, built in three different technologies. Our work has two sets of contributions. First, we illustrate the complexity of in-field DRAM error analysis and demonstrate the limitations of various widely-used methods and metrics. For example, we show that average error rates, errors per MB-hour and mean time between failures can provide volatile and unreliable results even after long periods of error logging, leading to incorrect conclusions about DRAM reliability. Second, we present formal statistical methods that overcome many of the limitations of the current approaches. The methods that we present are simple to understand and implement, reliable and widely accepted in the statistical community. Overall, our study alerts the community about the need to, firstly, question the current practice in quantifying DRAM reliability and, secondly, to select a proper analysis approach for future studies. Our strong recommendations are to focus on metrics with a practical value that could be easily related to system reliability, and to select methods that provide stable results, ideally supported with statistical significance. CCS CONCEPTS • Computer systems organization → Reliability; • Mathematics of computing → Probability and statistics.

show abstract

Machine Learning Models for GPU Error Prediction in a Large Scale HPC System

Cited by 61 publications

References 28 publications

Workload-Aware DRAM Error Prediction using Machine Learning

Workload-Aware DRAM Error Prediction using Machine Learning

Cost-Aware Prediction of Uncorrected DRAM Errors in the Field

DRAM errors in the field

Contact Info

Product

Resources

About