Inferring and asserting distributed system invariants

Abstract

Distributed systems are difficult to debug and understand. A key reason for this is distributed state, which is not easily accessible and must be pieced together from the states of the individual nodes in the system.We propose Dinv, an automatic approach to help developers of distributed systems uncover the runtime distributed state properties of their systems. Dinv uses static and dynamic program analyses to infer relations between variables at different nodes. For example, in a leader election algorithm, Dinv can relate the variable leader at different nodes to derive the invariant ∀ nodes i, j, leader i = leader j . This can increase the developer's confidence in the correctness of their system. The developer can also use Dinv to convert an inferred invariant into a distributed runtime assertion on distributed state.We applied Dinv to several popular distributed systems, such as etcd Raft, Hashicorp Serf, and Taipei-Torrent, which have between 1.7K and 144K LOC and are widely used. Dinv derived useful invariants for these systems, including invariants that capture the correctness of distributed routing strategies, leadership, and key hash distribution. We also used Dinv to assert correctness of the inferred etcd Raft invariants at runtime, using these asserts to detect injected silent bugs.

Monotone Precision and Recall Measures for Comparing Executions and Specifications of Dynamic Systems

“…Also, the works on the automated inference of models of behaviour including data attributes, time, and probabilities in their transitions [38,52,66,74,112] inspire an interesting future avenue for our research, i.e., to measure precision and recall of such extended models. Another context in which one can investigate the applicability and adaptation of our approach is that of the models of behaviour expressing distributed systems invariants [42]. Finally, one can design new quality measures that relate arbitrary numbers of behaviours (not just behaviours of a specification and its execution log), e.g., to establish a basis for comparing results of various process querying methods [79], models of behaviour that summarise traces at varying levels of abstraction [43], and different behavioural representations [83].…”

Visualizing Distributed System Executions

“…For two languages, Java and Go, we have versions of XVector that automatically instrument the source code using static analysis and a whitelist of known networking library calls. This procedure is described in more detail in our previous work on inferring distributed system invariants [46]. For the other languages, the developer must manually log information with XVector.…”

Towards Runtime Verification via Event Stream Processing in Cloud Computing Infrastructures

“…In this work, we identify patterns by manually inspecting the collected traces. In future work, we aim to develop algorithms to identify patterns using statistical analysis techniques, such as invariant analysis [17,34,20].…”