Verification-aided regression testing

Abstract

In this paper we present Verification-Aided Regression Testing (VART), a novel extension of regression testing that uses model checking to increase the fault revealing capability of existing test suites. The key idea in VART is to extend the use of test case executions from the conventional direct fault discovery to the generation of behavioral properties specific to the upgrade, by (i) automatically producing properties that are proved to hold for the base version of a program, (ii) automatically identifying and checking on the upgraded program only the properties that, according to the developers' intention, must be preserved by the upgrade, and (iii) reporting the faults and the corresponding counter-examples that are not revealed by the regression tests. Our empirical study on both open source and industrial software systems shows that VART automatically produces properties that increase the effectiveness of testing by automatically detecting faults unnoticed by the existing regression test suites.

VART: a tool for the automatic detection of regression faults

“…VART has been evaluated in terms of its capability to discover regression faults, its ability to provide sound results despite the usage of incomplete test suites, and its precision in the generation of the alarms [12].…”

“…Although changes are tested systematically, regression faults frequently remain silent until observed in the field. This is due to the extensiveness of regression test suites that, although designed to cover both the change and the presumably unchanged functionalities of the system, they may miss important corner cases affected by the change [12]. * This work has been partially supported by the H2020 Learn project, which has been funded under the ERC Consolidator Grant 2014 program (ERC Grant Agreement n. 646867).…”

“…To combine the power of static analysis together with the effectiveness of testing, we designed the Verification Aided Regression Testing (VART) technique [12], which exploits test cases to both automatically generate the correctness properties that must be checked and identify the properties that have been intentionally violated by the changes, and static analysis to both discover properties that provably hold before the change and detect properties that have been unintentionally violated by the change. Empirical results show that the combination of these two solutions can augment the classic regression testing process, which consists of running the available test cases after every change, with a fully automated verification process that can detect additional regression problems not revealed by the test cases, including faults that violate application-specific properties without producing any crash or exception.…”

A Synergistic Analysis Method for Explaining Failed Regression Tests

“…The concept of as correct as the previous version has become popular in the subfield of regression verification [22], [23]. Although these works do not directly focus on explaining failed regression tests as in this paper, they complement our work at the high level.…”

Automatic Failure Explanation in CPS Models

“…Failure explanation techniques analyze software failures in the attempt of producing information about failures and their causes. For instance, a few approaches combined mining and dynamic analysis in the context of componentbased and object-oriented applications to reveal [22] and explain failures [6,19,3]. These approaches are not however straightforwardly applicable to CPS models, since they exploit the discrete nature of component-based and object-oriented applications that is radically different from the data-flow oriented nature of CPS models, which include mixed-analog signals, hybrid (continuous and discrete) components, and a complex dynamics.…”