Debugging Distributed Systems with Why-Across-Time Provenance

Abstract

Systematically reasoning about the fine-grained causes of events in a real-world distributed system is challenging. Causality, from the distributed systems literature, can be used to compute the causal history of an arbitrary event in a distributed system, but the event's causal history is an overapproximation of the true causes. Data provenance, from the database literature, precisely describes why a particular tuple appears in the output of a relational query, but data provenance is limited to the domain of static relational databases. In this paper, we present wat-provenance: a novel form of provenance that provides the benefits of causality and data provenance. Given an arbitrary state machine, watprovenance describes why the state machine produces a particular output when given a particular input. This enables system developers to reason about the causes of events in real-world distributed systems. We observe that automatically extracting the wat-provenance of a state machine is often infeasible. Fortunately, many distributed systems components have simple interfaces from which a developer can directly specify wat-provenance using a technique we call wat-provenance specifications. Leveraging the theoretical foundations of wat-provenance, we implement a prototype distributed debugging framework called Watermelon.Returning to our example above, the why-provenance of the (Michael Jordan) tuple is the set {J 1 , J 2 } where J 1 = {(ecodd, jumpman), (jumpman, Michael Jordan)} J 2 = {(ecodd, mlpro), (mlpro, Michael Jordan)} J 1 and J 2 are minimal witnesses, so the why-provenance and minimal why-provenance of t = (Michael Jordan) are the same.