An Experimental Evaluation of Deliberate Unsoundness in a Static Program Analyzer

Abstract

Abstract. Many practical static analyzers are not completely sound by design. Their designers trade soundness to increase automation, improve performance, and reduce the number of false positives or the annotation overhead. However, the impact of such design decisions on the effectiveness of an analyzer is not well understood. This paper reports on the first systematic effort to document and evaluate the sources of unsoundness in a static analyzer. We developed a code instrumentation that reflects the sources of deliberate unsoundness in the .NET static analyzer Clousot and applied it to code from six open-source projects. We found that 33% of the instrumented methods were analyzed soundly. In the remaining methods, Clousot made unsound assumptions, which were violated in 2-26% of the methods during concrete executions. Manual inspection of these methods showed that no errors were missed due to an unsound assumption, which suggests that Clousot's unsoundness does not compromise its effectiveness. Our findings can guide users of static analyzers in using them fruitfully, and designers in finding good trade-offs.

What developers want and need from program analysis: an empirical study

“…Related to the types of issues that developers consider to be important are the potential sources of unsoundness in program analysis that can affect the detection of such issues. We listed the most common sources of unsoundness from program analysis research [28] and asked developers to rank up to five of them. During our initial interviews and the beta survey, we found that some developers were unfamiliar with the terminology used (though most were aware of the concepts).…”

“…By now, researchers have realized that soundness is commonly eschewed in practice [44]. In fact, there have even emerged approaches for measuring the unsoundness in a static analysis and evaluating its practical impact [28]. In industry, as it also becomes evident in Section 5, it is a well-established design decision to build program analyzers to be unsound in order to increase automation, improve performance, achieve modularity, and reduce the number of false positives or the annotation overhead.…”

Verification by abstract interpretation, soundness and abstract induction

“…This does not necessarily mean absolute correctness when putting restrictions (such as no parameter aliasing in Clousot [10] or soundness up to the first error with unpredictable effects in Astrée [23]), preferably machine checkable ones (such as the former restrictions on dynamic memory allocation in Astrée [23], now dropped).…”

Differentially testing soundness and precision of program analyzers

“…Unsoundness is ubiquitous in static analyzers [15], typically to intentionally favor other important qualities, such as precision or efficiency. A recent technique systematically documents and evaluates the sources of intentional unsoundness in a widely used, commercial static analyzer [16]. The experimental evaluation of this work sheds light on how often the unsoundness of the analyzer causes it to miss bugs.…”