Detection and Correction of Silent Data Corruption for Large-Scale High-Performance Computing

Abstract

Abstract-Faults have become the norm rather than the exception for high-end computing on clusters with 10s/100s of thousands of cores. Exacerbating this situation, some of these faults remain undetected, manifesting themselves as silent errors that corrupt memory while applications continue to operate and report incorrect results. This paper studies the potential for redundancy to both detect and correct soft errors in MPI message-passing applications. Our study investigates the challenges inherent to detecting soft errors within MPI application while providing transparent MPI redundancy. By assuming a model wherein corruption in application data manifests itself by producing differing MPI message data between replicas, we study the best suited protocols for detecting and correcting MPI data that is the result of corruption.To experimentally validate our proposed detection and correction protocols, we introduce RedMPI, an MPI library which resides in the MPI profiling layer. RedMPI is capable of both online detection and correction of soft errors that occur in MPI applications without requiring any modifications to the application source by utilizing either double or triple redundancy.Our results indicate that our most efficient consistency protocol can successfully protect applications experiencing even high rates of silent data corruption with runtime overheads between 0% and 30% as compared to unprotected applications without redundancy.Using our fault injector within RedMPI, we observe that even a single soft error can have profound effects on running applications, causing a cascading pattern of corruption in most cases causes that spreads to all other processes. RedMPI's protection has been shown to successfully mitigate the effects of soft errors while allowing applications to complete with correct results even in the face of errors.

A High-Level C++ Approach to Manage Local Errors, Asynchrony and Faults in an MPI Application

“…If a hard failure occurs it is not straight forward to continue the computation. The default way to handle such faults is a rollback to a previous checkpoint, which will be more and more expensive with increasing parallelism not only because of recomputation but also because of communication [13], [17]- [19]. In addition the communicator has to be re-established with replacement processes, or the application has to be repartitioned and/or load-balanced.…”

A backward/forward recovery approach for the preconditioned conjugate gradient method

“…The simplest technique is triple modular redundancy and voting [19], which induces a costly verification. For high-performance scientific applications, process replication (each process is equipped with a replica, and messages are quadruplicated) is proposed in the RedMPI library [20]. Elliot et al [21] combine partial redundancy and checkpointing, and confirm the benefit of dual and triple redundancy.…”

Detection of Silent Data Corruptions in Smoothed Particle Hydrodynamics Simulations

“…A high precision and a high recall indicate both few false-positives and a good detection rate, respectively. In general, detectors that either employ full replication of entire applications [27] or selective replication of parts of an application [10] offer the highest precision and recall. However, they are often prohibitively expensive in terms of additional required computing resources and time.…”