Implementing efficient message logging protocols as MPI application extensions

Abstract

Message logging protocols are enablers of local rollback, a more efficient alternative to global rollback, for fault tolerant MPI applications. Until now, message logging MPI implementations have incurred the overheads of a redesign and redeployment of an MPI library, as well as continued performance penalties across various kernels. Successful research efforts for message logging implementations do exist, but not a single one of them can be easily deployed today by more than a few experts. In contrast, in this work we build efficient message logging capabilities on top of an MPI library with no message logging capabilities; we do so for two different HPC kernels, one with a global exchange pattern (CG), and one with a neighbourhood exchange pattern (LULESH). While our library of choice ULFM detects failure and recovers MPI communicators, we build on that to then restore the intra-and inter-process data consistency of both applications. This task turns out to be challenging, and we present the methodology for doing so in this work. In the end, we achieve message logging capabilities for each kernel, without the need for an actual message logging runtime underneath. On the performance side, we match state-of-the-art solutions and (a) eliminate event logging and the event logger component altogether, and (b) design a hybrid protocol, which gracefully shifts between global and local rollback, depending on the available payload logging memory. Such a hybrid protocol between local and global rollback has not been previously proposed to our knowledge. Our extensions span a few hundred lines of code for each kernel, are open-sourced, and enable local and global rollback after process failure.

Power Log’n’Roll: Power-Efficient Localized Rollback for MPI Applications Using Message Logging Protocols

“…We find deploying and experimenting with the VProtocol challenging. The difficulties in experimenting with message logging runtimes in HPC today have driven us to implement a working and compact message logging protocol as application extensions [22]. This opens up the path for experimenting with further features, such as power savings, as demonstrated in this work.…”

“…Message logging protocols have been studied widely in the past, and found various uses; one of them is the ability to forge combinations of uncoordinated checkpointing and message logging, which reduces the overheads of checkpointing. However, we here use message logging protocols in another popular direction [20], [22] -in order to enable local rollback.…”

“…We find the best available implementation [20], Open MPI, challenging for enabling message logging protocols (we had difficulties both with payload logging via the Vprotocol, as well as with the event logger). Our work therefore implements message logging as application extensions [22], instead of using message logging runtimes. To our knowledge, our design is unique in the message logging landscape.…”

“…We have validated our message logging extensions by checking the application data integrity at various postfailure stages [22].…”

“…It has been shown (e.g. [20], [22]) that logs only need to be kept for recent enough time frames, regardless of execution duration. In the latter contribution, we have measured the overheads of message logs to the memory footprint at less than 1% for two of the tested kernels.…”