Cooperative verifier-based testing with CoVeriTest

Abstract

Testing is a widely applied technique to evaluate software quality, and coverage criteria are often used to assess the adequacy of a generated test suite. However, manually constructing an adequate test suite is typically too expensive, and numerous techniques for automatic test-suite generation were proposed. All of them come with different strengths. To build stronger test-generation tools, different techniques should be combined. In this paper, we study cooperative combinations of verification approaches for test generation, which exchange high-level information. We present CoVeriTest, a hybrid technique for test-suite generation. CoVeriTest iteratively applies different conditional model checkers and allows users to adjust the level of cooperation and to configure individual time limits for each conditional model checker. In our experiments, we systematically study different CoVeriTest cooperation setups, which either use combinations of explicit-state model checking and predicate abstraction, or bounded model checking and symbolic execution. A comparison with state-of-the-art test-generation tools reveals that CoVeriTest achieves higher coverage for many programs (about 15%).

Gray-Box Fuzzing via Gradient Descent and Boolean Expression Coverage

“…For the presentation purposes, "ReachSafety" and "SoftwareSystems" substrings in the family names are shortened to "rs" and "ss", respectively, in the rest of this section. For comparison, we used three best-scoring tools 3 from Test-Comp 2023, namely FuSeBMC [2], VeriFuzz [21], and CoVeriTest [6], in the versions in which they entered Test-Comp 2023. To obtain reproducible results, we asked the organizer of Test-Comp to evaluate Fizzer on the official infrastructure of Test-Comp and compare the obtained results with the official results of Test-Comp 2023.…”

“…We evaluated the tests generated by Fizzer using the competition infrastructure which measures the achieved branch coverage. The results presented in Section 7 show that our tool is competitive with the top-ranking tools of Test-Comp 2023, namely FuSeBMC [2], VeriFuzz [21], and CoVeriTest [6]. Note that our tool is a pure gray-box fuzzer while FuSeBMC and VeriFuzz combine dynamic analysis with static analyses like symbolic execution and bounded model checking.…”

“…Ver-iFuzz [21] is built on top of afl and an engine based on Coverage Guided Fuzzing, combined with the bounded model checker cbmc and the prism framework. CoVeriTest [6] combines several model checkers.…”