Statically-directed dynamic automated test generation

Abstract

We present a new technique for exploiting static analysis to guide dynamic automated test generation for binary programs, prioritizing the paths to be explored. Our technique is a three-stage process, which alternates dynamic and static analysis. In the first stage, we run dynamic analysis with a small number of seed tests to resolve indirect jumps in the binary code and build a visibly pushdown automaton (VPA) reflecting the global control-flow of the program. Further, we augment the computed VPA with statically computable jumps not executed by the seed tests. In the second stage, we apply static analysis to the inferred automaton to find potential vulnerabilities, i.e., targets for the dynamic analysis. In the third stage, we use the results of the prior phases to assign weights to VPA edges. Our symbolic-execution based automated test generation tool then uses the weighted shortest-path lengths in the VPA to direct its exploration to the target potential vulnerabilities. Preliminary experiments on a suite of benchmarks extracted from real applications show that static analysis allows exploration to reach vulnerabilities it otherwise would not, and the generated test inputs prove that the static warnings indicate true positives.

Covrig: a framework for the analysis of code, test, and coverage evolution in real software

“…requiring no program execution). We suspect that the main reason behind the scarcity of studies focusing on dynamic metrics lies in the difficulty of running multiple software versions, 1 especially since doing so involves evolving dependencies and unstable (including non-compilable) versions. For example, prior work [41] cites the manual effort and the long time needed to run different revisions as the reason for reporting dynamic measurements for only a small number of versions.…”

“…For example, the research community has invested a lot of effort in designing techniques for improving the testing of software patches, ranging from test suite prioritisation and selection algorithms [11,30,35] to program analysis techniques for test case generation and bug finding [1,2,20,21,27,28,36,40] to methods for surviving errors introduced by patches at runtime [14]. Many of these techniques depend on the existence of a manual test suite, sometimes requiring the availability of a test exercising the patch [24,37], sometimes making assumptions about the stability of program coverage or external behaviour over time [14,29], other times using it as a starting point for exploration [10,16,22,39], and often times employing it as a baseline for comparison [3,6,9,26].…”

Alternating Control Flow Reconstruction

“…In this CFG, control flow can even reach addresses such as 1 and 2, which lie between existing instructions. 1 Moreover, the imprecision of over-approximating the jump edges propagates: through the spurious edge (6, assume (x = 22), 22), the second indirect jump can be reached, leading to a self-loop at location 22.…”

“…In recent work, Babic et al [1] construct a CFG by folding a set of concrete traces and exploring the unexecuted branches of conditional jumps. In our own framework, this corresponds to using trace replay with a trivial static analysis that only knows a single state and thus always enables both branches of conditional jumps.…”

High-Coverage Symbolic Patch Testing

“…Synthesizing inputs which cover a target is an essential problem in automated test generation and debugging [1,16,27,29,30,32]. While we borrow ideas from the state of the art in these areas, and combine symbolic execution, static analysis and various heuristics, our approach differs by treating the task as an optimization problem with the goal of exploring paths that minimize the estimated distance to the target.…”