On the effect of test-suite reduction on automatically generated model-based tests

Abstract

Model checking techniques can be successfully employed as a test-case generation technique to generate tests from formal models. The number of tests-cases produced, however, is typically large for complex coverage criteria such as MC/DC. Test-suite reduction can provide us with a smaller set of test-cases that preserve the original coverage-often a dramatically smaller set. Nevertheless, one potential drawback with test-suite reduction is that this might affect the quality of the test-suite in terms of fault finding. Previous empirical studies provide conflicting evidence on this issue. To further investigate the problem and determine its effect when testing implementations derived from formal models of software we performed an experiment using a large case example of a Flight Guidance System, generated reduced test-suites for a variety of structural coverage criteria while preserving coverage, and recorded their fault finding effectiveness. Our results indicate that the size of the specification based test-suites can be dramatically reduced and that the fault detection of the reduced test-suites is adversely affected. In this report we describe our experiment, analyze the results, and discuss the implications for testing based on formal specifications.

Prioritizing test cases with string distances

“…But Wong in Wong et al (1999) uses different case studies and does not notice significant fault detection loss in reduced test suites. More recently, Heimdahl stated significant fault-finding loss in other experiments (Heimdahl and Devaraj 2007).…”

“…Although many works rely on the availability of code coverage information (Harrold et al 1993;Jones and Harrold 2003), some other works do not require a code; they are based on coverage of a specification of the program under test (Heimdahl and Devaraj 2007), or coverage of the input syntax (Hennessy and Power 2005).…”

Model-Based Filtering of Combinatorial Test Suites

“…The approach presented in [14] generates test suites from a model and traps properties corresponding to structural coverage criteria. An algorithm is then executed on the resulting test suite to generate a reduced test suite having the same coverage than the original one.…”

Model Based Automatic Test Research and Application

“…Take a look in model-based self-testing of the general process. first, Modeling Second, Generate Expected Input Third, Generating Expected Output Fourth, Run the Test Fifth ,comparing the actual output and expected output, and the icon (5) to decide the next step (whether to change models, generate more tests, or to stop testing, estimate software reliability (quality))…”