Pervasive detection of process races in deployed systems

Abstract

Process races occur when multiple processes access shared operating system resources, such as files, without proper synchronization. We present the first study of real process races and the first system designed to detect them. Our study of hundreds of applications shows that process races are numerous, difficult to debug, and a real threat to reliability. To address this problem, we created RACEPRO, a system for automatically detecting these races. RACEPRO checks deployed systems in-vivo by recording live executions then deterministically replaying and checking them later. This approach increases checking coverage beyond the configurations or executions covered by software vendors or beta testing sites. RACEPRO records multiple processes, detects races in the recording among system calls that may concurrently access shared kernel objects, then tries different execution orderings of such system calls to determine which races are harmful and result in failures. To simplify race detection, RACEPRO models under-specified system calls based on load and store micro-operations. To reduce false positives and negatives, RACEPRO uses a replay and go-live mechanism to distill harmful races from benign ones. We have implemented RACEPRO in Linux, shown that it imposes only modest recording overhead, and used it to detect a number of previously unknown bugs in real applications caused by process races.

P axos made transparent

“…First, CRANE can be leveraged by other replication concepts (e.g., byzantine fault tolerance [22,38]) and record-replay [39,41,46] because they also suffer from nondeterminism. Second, promising results in REPFRAME [30] have shown that CRANE's transparent replication architecture can enable multiple types of program analysis tools within one execution, making a server program enjoy benefits of multiple analyses.…”

An observable and controllable testing framework for modern systems

“…However, this approach can be both ineffective and inefficient as it cannot determine the location at which to issue an interrupt. Laadan et al [10] note the importance of process races and design a technique to detect and replay them. However, their technique can produce false negatives, and cause replay divergence failures in which the actual system environment does not match the replayed execution.…”

Transparent mutable replay for multicore debugging and patch validation

“…Our previous work on Scribe [16] replays a recorded application execution until a specified point, and then transitions to live execution instead of replaying the rest of the log. Our previous work on Racepro [17] detects process races due to dependencies in the ordering of system calls by recording an application execution to a log, identifying a pair of system calls that may be racy, truncating the log at the occurrence of the pair of system calls, inverting their order, and then replaying the truncated log with the reordered system calls to detect process races. However, Racepro only supports changes that reorder system calls and does not support changes in the middle of replay.…”