Using Model Checking to Generate Tests from Requirements Specifications

Gargantini, Angelo; Heitmeyer, Constance L.

doi:10.1007/3-540-48166-4_10

Cited by 105 publications

(73 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Examples include heuristic search methods [38] and approaches based on reachability [27], [3], [4]. Such techniques have advanced to the point where they can be effectively applied to production systems.…”

Section: Methodsmentioning

confidence: 99%

“…Most studies concerning automatic test generation for structural coverage criteria are focused on how to generate tests quickly and/or improve coverage [27], [3]. Comparisons of the fault-finding effectiveness of the resulting test suites against other methods of test generation are few.…”

Section: Related Workmentioning

confidence: 99%

“…For our directed test generation approach, we used counterexample-based test generation to generate tests satisfying the four coverage criteria [27], [3]. In this approach, each coverage obligation is encoded as a temporal logic formula and a model checker can be used to detect a counterexample (test case) illustrating how the coverage obligation can be covered.…”

Section: Test Data Generationmentioning

confidence: 99%

See 2 more Smart Citations

The Risks of Coverage-Directed Test Case Generation

Gay

Staats

Whalen

et al. 2015

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

A number of structural coverage criteria have been proposed to measure the adequacy of testing efforts. In the avionics and other critical systems domains, test suites satisfying structural coverage criteria are mandated by standards. With the advent of powerful automated test generation tools, it is tempting to simply generate test inputs to satisfy these structural coverage criteria. However, while techniques to produce coverage-providing tests are well established, the effectiveness of such approaches in terms of fault detection ability has not been adequately studied. In this work, we evaluate the effectiveness of test suites generated to satisfy four coverage criteria through counterexample-based test generation and a random generation approach-where tests are randomly generated until coverage is achieved-contrasted against purely random test suites of equal size. Our results yield three key conclusions. First, coverage criteria satisfaction alone can be a poor indication of fault finding effectiveness, with inconsistent results between the seven case examples (and random test suites of equal size often providing similar-or even higher-levels of fault finding). Second, the use of structural coverage as a supplement-rather than a target-for test generation can have a positive impact, with random test suites reduced to a coverage-providing subset detecting up to 13.5% more faults than test suites generated specifically to achieve coverage. Finally, Observable MC/DC, a criterion designed to account for program structure and the selection of the test oracle, can-in part-address the failings of traditional structural coverage criteria, allowing for the generation of test suites achieving higher levels of fault detection than random test suites of equal size. These observations point to risks inherent in the increase in test automation in critical systems, and the need for more research in how coverage criteria, test generation approaches, the test oracle used, and system structure jointly influence test effectiveness.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Test Data Generationmentioning

confidence: 99%

See 1 more Smart Citation

The Risks of Coverage-Directed Test Case Generation

Gay

Staats

Whalen

et al. 2015

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

show abstract

“…Second, model checkers can significantly reduce the time and effort needed to check for security and safety violations. Third, tests which are systematically generated from a formal model (see, e.g., [9], can eliminate the redundancy often found in conventional software testing, thus reducing the number of tests and the time needed for testing.…”

Section: (2) Minimizing Time-to-market For Apps and Other Consumer Omentioning

confidence: 99%

What Happened to Formal Methods for Security?

Schaffer

Voas

2016

Computer

View full text Add to dashboard Cite

We asked 7 experts 7 questions to find out what has occurred recently in terms of applying formal methods (FM) to security-centric, cyber problems. We are continually reminded of the 1996 paper by Tony Hoare “How did Software Get So Reliable Without Proof?” [1] In that vein, how did we get so insecure with proof? Given daily press announcements concerning new malware, data breaches, and privacy loss, is FM still relevant or was it ever? Our experts answered with unique personal insights. We were curious as to whether this successful methodology in “safety-critical” has succeeded as well for today’s “build it, hack it, patch it” mindset. Our experts were John McLean (Naval Research Labs), Paul Black (National Institute of Standards and Technology), Karl Levitt (University of California at Davis), Joseph Williams (CloudEconomist.Com), Connie Heitmeyer (Naval Research Labs), Eugene Spafford (Purdue University), and Joseph Kiniry (Galois, Inc.). The questions and responses follow.

show abstract

“…A classical technique for model-based test generation exploits the capability of model checkers to produce counterexamples [10,12]. If a test predicate can be expressed as a CTL/LTL formula over the model states, then a test suite covering the test goals corresponding to a desired coverage criterion can be generated as follows.…”

Section: Definition 5 (Test Predicate)mentioning

confidence: 99%

An Abstraction Technique for Testing Decomposable Systems by Model Checking

2014

Self Cite

View full text Add to dashboard Cite

Abstract. Test generation by model checking exploits the capability of model checkers to return counterexamples upon property violations. The approach suffers from the state explosion problem of model checking. For property verification, different abstraction techniques have been proposed to tackle this problem. However, such techniques are not always suitable for test generation. In this paper we focus on Decomposable by Dependency Asynchronous Parallel (DDAP) systems, composed by several subsystems running in parallel and connected together in a way that the inputs of one subsystem are provided by another subsystem. We propose a test generation approach for DDAP systems based on a decompositional abstraction that considers one subsystem at a time. It builds tests for the single subsystems and combines them later in order to obtain a global system test. Such approach avoids the exponential increase of the test generation time and memory consumption. The approach is proved to be sound, but not complete.

show abstract

Using Model Checking to Generate Tests from Requirements Specifications

Cited by 105 publications

References 16 publications

The Risks of Coverage-Directed Test Case Generation

The Risks of Coverage-Directed Test Case Generation

What Happened to Formal Methods for Security?

An Abstraction Technique for Testing Decomposable Systems by Model Checking

Contact Info

Product

Resources

About