Measuring enforcement windows with symbolic trace interpretation: what well-behaved programs say

Abstract

A static analysis design is sufficient if it can prove the property of interest with an acceptable number of false alarms. Ultimately, the only way to confirm that an analysis design is sufficient is to implement it and run it on real-world programs. If the evaluation shows that the design is insufficient, the designer must return to the drawing board and repeat the process-wasting expensive implementation effort over and over again. In this paper, we make the observation that there is a minimal range of code needed to prove a property of interest under an ideal static analysis; we call such a range of code a validation scope. Armed with this observation, we create a dynamic measurement framework that quantifies validation scopes and thus enables designers to rule out insufficient designs at lower cost. A novel attribute of our framework is the ability to model aspects of static reasoning using dynamic execution measurements. To evaluate the flexibility of our framework, we instantiate it on an example property-null dereference errors-and measure validation scopes on real-world programs. We use a broad range of metrics that capture the difficulty of analyzing programs along varying dimensions. We also examine how validation scopes evolve as developers fix null dereference errors and as code matures. We find that bug fixes shorten validation scopes, that longer validation scopes are more likely to be buggy, and that overall validation scopes are remarkably stable as programs evolve.

Trace Typing: An Approach for Evaluating Retrofitted Type Systems (Extended Version)

“…Other Uses of Traces Coughlin et al [10] use dynamic traces to identify the spans between program safety checks and data uses guarded by those checks, e.g., a null check followed by a dereference. These measurements can inform the design of static analyses, e.g., by indicating whether or not an analysis should be interprocedural.…”

“…In fact, TypeScript has long supported overloaded functions for expressing intersection types. 10 The typescript benchmark stands out from the others as it was originally implemented in TypeScript, and hence we would expect it to be mostly well typed. Our results confirm this: the error count is only 290 in 8139 (JavaScript) lines of code for the least precise configuration, a much lower error rate than the other programs.…”