Leveraging existing instrumentation to automatically infer invariant-constrained models

Abstract

Computer systems are often difficult to debug and understand. A common way of gaining insight into system behavior is to inspect execution logs and documentation. Unfortunately, manual inspection of logs is an arduous process and documentation is often incomplete and out of sync with the implementation. This paper presents Synoptic, a tool that helps developers by inferring a concise and accurate system model. Unlike most related work, Synoptic does not require developer-written scenarios, specifications, negative execution examples, or other complex user input. Synoptic processes the logs most systems already produce and requires developers only to specify a set of regular expressions for parsing the logs.Synoptic has two unique features. First, the model it produces satisfies three kinds of temporal invariants mined from the logs, improving accuracy over related approaches. Second, Synoptic uses refinement and coarsening to explore the space of models. This improves model efficiency and precision, compared to using just one approach.In this paper, we formally prove that Synoptic always produces a model that satisfies exactly the temporal invariants mined from the log, and we argue that it does so efficiently. We empirically evaluate Synoptic through two user experience studies, one with a developer of a large, real-world system and another with 45 students in a distributed systems course. Developers used Synoptic-generated models to verify known bugs, diagnose new bugs, and increase their confidence in the correctness of their systems. None of the developers in our evaluation had a background in formal methods but were able to easily use Synoptic and detect implementation bugs in as little as a few minutes.

Automatic mining of specifications from invocation traces and method invariants

“…Results that generalize to poorly-understood libraries. 4. Enhancements to an existing third-party technique [16], further corroborating our hypothesis that using inferred invariants during model construction yields higher-quality models.…”

“…Such a specification should, for example, assert that communication libraries (e.g., Apache's [1] libraries) require a connection to be established before data is sent. Numerous techniques have been proposed to that end [3,4,10,15,19,20,32,33,41,46]. These techniques either (1) infer finite state machine (FSM) models that match the observed executions [3,4,9,20,32,38,44], or (2) identify the declarative class and method invariants by considering a library's state (i.e., its internal variables) [10,19,45].…”

“…While the artifacts produced by these techniques can assist development tasks (e.g., test case generation [14,33] and debugging [4]), their utility critically depends on how close they are to the "true model." This, in turn, depends on the quality of the input executions, which are inherently partial.…”

Using contexts to extract models from code

“…The approach presented in [5] (which extends the work described in [6]) uses the CSight tool to produce Communicating Finite State Machines (CFSM) from logs of execution based on regular expressions provided by users. They deal with logs of concurrent systems and require that vector timestamps be inserted in the traces to determine the partial order of the concurrent events.…”

Model Counting for Complex Data Structures

“…Usage profiles can be specified based on physical phenomena, known sensor parameters or other domain specific knowledge about the program and its deployment context. They can also be built automatically based on observed data from past usages of the program [19,5].…”