Model Based Testing with Logical Properties versus State Machines

Abstract

Model-based testing of single functions is usually based on logical properties as specification. In practice it appears to be rather hard to develop a sufficiently strong set of properties to spot all errors. For model-based testing of state based systems one usually employs a state machine as model and a conformance relation between this specification and the software under test. For (abstract) data types we can use both ways of model-based testing. In this paper we compare the specification effort required to make a model that is able to find issues and the number of tests needed to find issues for some well-known data types. Our examples show that it can be easier to write state based specifications. Moreover, state based testing of data types finds more implementation issues and is very efficient.

Property-Based Testing: Climbing the Stairway to Verification

“…It is often easier to use a model rather than a set of properties to define the behaviour of functions [43]; our experiments concurred. For example, functions in the C library can be readily modeled in terms of Haskell library functions.…”

“…We instead test functions against a full executable specification that models the functions. This is conceptually similar to model-based testing [43] (also see Section 8). The functional specification is most akin to the functional notions of model paradigm as classified in the work by Utting et al [64, ğ 3.3].…”

“…While PBT is effective, it is not universally applicableÐin practice, it is often hard to describe the full behaviour of a system solely in terms of logical properties [43]. In the context of formal verification, however, using a functional specification to describe the behaviour of a systems is very common.…”

“…This state-based model for specifying the behaviours of systems is a very common paradigm in the world of modelbased testing (MBT) [33,43,64]. However, as mentioned in Section 1, Cogent's purely functional semantics provides a simple formal model of a program's behaviour; specifically, it enables reasoning about programs using equational principles.…”

“…Instead of testing logical properties about the system, which is what PBT is typically designed for, we test the refinement relation between the implementation and the specification. Using logical properties to describe the behaviour of a system has been criticised for its practicability [43], especially if the full functional correctness of the system is desired. Instead, high-level properties of a system can be proved on top of its functional specification.…”