Optimizing customized program coverage

Abstract

Program coverage is used across many stages of software development. While common during testing, program coverage has also found use outside the test lab, in production software. However, production software has stricter requirements on run-time overheads, and may limit possible program instrumentation. Thus, optimizing the placement of probes to gather program coverage is important. We introduce and study the problem of customized program coverage optimization. We generalize previous work that optimizes for complete coverage instrumentation with a system that adapts optimization to customizable program coverage requirements. Specifically, our system allows a user to specify desired coverage locations and to limit legal instrumentation locations. We prove that the problem of determining optimal coverage probes is NP-hard, and we present a solution based on mixed integer linear programming. Due to the computational complexity of the problem, we also provide two practical approximation approaches. We evaluate the effectiveness of our approximations across a diverse set of benchmarks, and show that our techniques can substantially reduce instrumentation while allowing the user immense freedom in defining coverage requirements. When naïve instrumentation is dense or expensive, our optimizations succeed in lowering execution time overheads. CCS Concepts •Software and its engineering → Software testing and debugging; Software post-development issues; Software performance; •Mathematics of computing → Integer programming;

INSTRIM: Lightweight Instrumentation for Coverage-guided Fuzzing

“…Ohmann et al [6] consider binarized coverage and modeled the problem using integer linear programming and proved it is NP-hard. They proposed several approximation algorithms, which still need to mark about a half of the total vertices.…”

“…AFL can optionally instrument a random subset of basic blocks, but its impact on coverage information requires careful evaluation. Moreover, most existing techniques to reduce instrumentation overhead [2], [8], [6] either rely on simple heuristics or become inefficient when being directly applied to coverage-guided fuzzing. Hence, the core research questions we would like to investigate are: What techniques can be applied to reduce the cost of instrumentation?…”

A Comprehensive Technological Survey on Dependable Self-Managing CPS: The Decade of Researches on Correctness and Dependability

“…2) The coverage of testing is an NP-hard problem [71] which implies that no system can be completely correct.…”

Log20

“…Larus later extended this method by adding latency information and the ability to trace inter-procedurally [16]. Ohmann et al [24] formalized the problem of customized program coverage, that is to instrument the program while respecting a certain set of constraints in order to generate coverage information for a given set of program points. Their practical methods used static analysis and approximation algorithms to compute placements that minimized the runtime of instrumentation.…”