Using model checking to generate tests from requirements specifications

Abstract

Recently, many formal methods, such as the SCR (Software Cost Reduction) requirements method, have been proposed for improving the quality of software specifications. Although improved specifications are valuable, the ultimate objective of software development is to produce software that satisfies its requirements. To evaluate the correctness of a software implementation, one can apply black-box testing to determine whether the implementation, given a sequence of system inputs, produces the correct system outputs. This paper describes a specification-based method for constructing a suite of test sequences, where a test sequence is a sequence of inputs and outputs for testing a software implementation. The test sequences are derived from a tabular SCR requirements specification containing diverse data types, i.e., integer, boolean, and enumerated types. Prom the functions defined in the SCR specification, the method forms a collection of predicates called branches, which "cover" all possible software behaviors described by the specification. Based on these predicates, the method then derives a suite of test sequences by using a model checker's ability to construct counterexamples. The paper presents the results of applying our method to four specifications, including a sizable component of a contractor specification of a real system.

On the Danger of Coverage Directed Test Case Generation

“…Most studies concerning automatic test generation for structural coverage criteria are focused on how to generate tests quickly and/or improve coverage [19,3]. Comparisons of the fault-finding effectiveness of the resulting test suites against other methods of test generation are few.…”

“…Examples include heuristic search methods and approaches based on reachability [19,3,4]. Such techniques have advanced to the point where they can be effectively applied to real-world avionics systems.…”

“…For our directed test generation approach, we used counterexample-based test generation to generate tests satisfying branch and MC/DC coverage [19,3]. In this approach each coverage obligation is encoded as a temporal logic formula and the model checker can be used to detect a counterexample (test case) illustrating how the coverage obligation can be covered.…”

Testing with model checkers: a survey

“…As will be shown in Sections 4 and 5, a popular approach is to automatically create never-claims based on coverage criteria. These never-claims are called trap properties [32], and for each item that should be covered one trap property is generated. A test purpose is not necessarily feasible, but fortunately infeasible test purposes are not a problem, because the never-claim for an infeasible test purpose simply results in no counterexample.…”

Testing Systems of Concurrent Black-Boxes—An Automata-Theoretic and Decompositional Approach

“…Most work (e.g., [6,10,13]) stems from the spirit of specification-based testing, and utilizes model-checkers' capabilities of generating counter-examples from a system's specification to produce test-cases against an implementation. This approach typically works at the unit level and lacks a "control" over the generated test-cases, since, unlike our technique, it does not have an overall and analytical characterization over all the useful (i.e., has the potential to recover a global bad behavior) test sequences.…”