Regression Testing Process Improvement for Specification Evolution of Real-World Protocol Software

Abstract

Model-based testing helps test engineers automate their testing tasks so that they can be more costeffective. When the model is changed due to the evolution of the specification, it is important to maintain the test suites up to date for regression testing. A complete regeneration of the whole test suite from the new model, although inefficient, is still frequently used in practice. To handle specification evolution effectively, we propose a test case reusability analysis technique to identify reusable test cases of the original test suite based on graph analysis, so that we can generate new test cases to cover only the change-related parts of the new model. Our experiment on four large protocol document testing projects shows that the technique can significantly reduce regression testing time when compared with complete regeneration of the test suites.

Selective Test Generation Method for Evolving Critical Systems

“…We now compare the SeTGaM method proposed in this paper with other test generation techniques for managing evolutions: retest-all [7] and regenerate-all [5]. Retest-all is a commonly used regression testing technique which consists in re-executing all tests from an old test suite on the evolved model and then generate tests for uncovered requirements' behaviors.…”

“…When facing large models from industrial case studies, regenerating all the tests can be costly. For example, as stated by Jiang et al in [5] complete regeneration of the test suite for the model of Microsoft protocol documentation testing project may take hours or even a full day. Then, test engineers must check all non affected parts, which may take several days, up to several weeks, depending on the model scale.…”

“…The model's scale is very important factor for the automated test generation time [5]. For huge industrial critical systems, such as Air Traffic Managers, or Smart Card environments, such as GlobalPlatform (GP) [8], [9] having hundreds of different behaviors and different variables, it may takes several days and even more.…”

How well does test case prioritization integrate with statistical fault localization?

“…Our first--hand experience [24] on testing Microsoft protocols shows that executing the entire regression test suite for one protocol testing project can take more than a whole day. The continuous integration process stresses on fast feedback to the developers upon their code commit.…”

Automatically repairing test cases for evolving method declarations

“…Xie et al developed some techniques to automatically identify and correct graphical objects affected by GUI changes [20], [8]. Jiang et al and Memon addressed generic problems like correcting sequences of test scripts, for instance, pressing buttons or invoking methods [15], [12]. Such techniques require accurate system specifications or oracles.…”