A Checkpoint/Restart Scheme for CUDA Programs with Complex Computation States

Abstract

Checkpoint/restart has been an effective mechanism to achieve fault tolerance for many long-running scientific applications. The common approach is to save computation states in memory and secondary storage for execution resumption. However, as the GPU plays a much bigger role in high performance computing, there is no effective checkpoint/restart scheme yet due to the difficulty of the GPU computation state handling. This paper proposes an application-level checkpoint/restart scheme to save and restore GPU computation states in annotated user programs. A pre-compiler and run-time support module are developed to construct and save states in CPU system memory dynamically, whereas secondary storage can be utilized for scalability and long-term fault tolerance. CUDA programs with complicated computation states are supported. State-related variables dissipated in various memory units are collected. Both stack and heap are duplicated at application level for state construction. Experimental results have demonstrated the effectiveness of the proposed scheme.

CRAC: An automatic assistant compiler of checkpoint/restart for OpenCL program

“…Despite this, fine-grained migrations in the GPU kernel have also been proposed several times. In this article, the design of Jiang et al 26 was summarized in detail, where the source code was divided into multiple parts according to the given checkpoint. Until the computation was finished, the kernel would not stop being launched circularly.…”