Software testing based on formal specifications: a theory and a tool

Abstract

This paper addresses the problem of constructing test data sets from formal specifications. Starting from a notion of an ideal exhaustive test data set which is derived from the notion of satisfaction of the formal specification, it is shown how to select by refinements a practicable test set, i.e. computable, not rejecting correct programs (unbiased), and accepting only correct programs (valid), assuming some hypotheses. The hypotheses play an important role: they formalize common test practices and they express the gap between the success of the test and correctness ; the size of the test set depends on the strength of the hypotheses.The paper shows an application of this theory in the case of algebraic specifications and presents the actual procedures used to mechanically produce such test sets, using Horn clause logic. These procedures are embedded in an interactive system which, given some general hypotheses schemes and an algebraic specification, produces a test set and the corresponding hypotheses.

Formal Methods and Testing: Hypotheses, and Correctness Approximations

“…In most cases, the entity under test is a system, and the "challenging difficulties" are inputs, or sequences of inputs, aiming at revealing some dysfunctions [4], [8], [12] [13], [15], etc. In such cases, formal descriptions of the system are mainly used as guidelines for the selection of (sequences of) inputs and for the verdict.…”

“…This relation is usually defined on a semantic domain common to implementations and specifications (i.e. there is some domain D such that sat Õ D¥D) [4], [9], [20], but in some cases they may be different (sat Õ D1¥D2) [9].…”

“…Historically the above framework has been developed for algebraic specifications [4], [22]. Test data are just instantiated axioms of the specification and test experiments consist in their evaluation by the SUT to check that they are satisfied.…”

“…weaker uniformity hypotheses. In the case of positive conditional axioms, this method has been automated by the LOFT constraint solver [4].…”

“…They provide some approximation of correctness that is correlated to these hypotheses. In this talk we recall the notions of testability hypotheses and selection hypotheses that were introduced in [4], and we show how they have been used or could be used on various kinds of formal methods. We also address the issues of observation and control of the system under test.…”