Generation of test sequences from formal specifications: GSM 11‐11 standard case study

Abstract

This paper presents the results of a case study on generating test cases for a fragment of the smart card GSM 11-11 standard. The generation method is based on an original approach using the B notation and techniques of constraint logic programming with sets. The GSM 11-11 technical specifications were formalized with the B notation. From this B specification, a system of constraints was derived, equivalent to this formal model. Using a set constraint solver, boundary states were computed and test cases were obtained by traversing the constrained reachability graph of the specifications. The purpose of this project was to evaluate the contribution of this testing environment, called B-TESTING-TOOLS, in an industrial process on a real life-size application, by comparing the generated test sequences with the already used and high-quality manually-designed tests. This comparison enabled us to validate our approach and showed its effectiveness in the validation process of critical applications: the case study gives a wide coverage (about 85%) of the generated tests compared to the pre-existing tests and a saving of 30% in test design time.the specifications are implemented exactly as expected. In application domains of critical software, i.e. software, avionics, railway signalling, automotive control and management systems, secure smart card applications, etc., more than 50% of the development is devoted to the validation of the software product. Testing is performed by applying test cases to an implementation under test, by making observations during the execution of the tests, and by subsequently assigning a verdict on the correct execution of the implementation. The design of black-box tests in industry is seldom supported by tools and relies on the expertise of validation engineers. Consequently, there is considerable subjectivity exercised in created test sets. The manual design of functional black-box tests is generally based around an informal review of the system specification.The need to offer better methods and tools for functional black-box testing has given rise to a large amount of research on generating tests from formal specifications, e.g. [1][2][3]. Formal methods of specification, particularly model-oriented notations such Z [4], VDM [5] and B [6], allow a high-level abstract formalization of the expected behaviour of the system under test. These notations are well suited for test generation because the expressiveness of set-oriented logic constructs and the definition of an explicit model help both test case generation and oracle synthesis. Thus, these formal notations are the basis of various proposals to more or less automatically generate tests from the formal model, see for example [7][8][9][10][11].In [12][13][14], we presented a new method for automated test generation from B abstract machines using constraint logic programming [15], called B-TESTING-TOOLS. From the B formal model of the functional requirements of the system to be tested, an equivalent system of constraints is derived and, th...

An empirical study of test generation with BETA

“…BZ-TT/LTG was evaluated in academy and industry. In [31], BZ-TT/LTG was applied to an industrial case study in the Smart Card domain and its results were compared with manual tests. The assessment showed that the tests generated by BZ-TT/LTG covered 80 % of the manual tests and saved 30 % of the test design time.…”

Improving Verification & Validation in the Medical Device Domain

“…Verification and Validation (V&V) are amongst the most important activities in the software development lifecycle and consume up to 50% of project development time [1], [2] and up to 50% of the total cost [3]. The concepts of V&V emerged in the late 1960s and 1970s as the use of software in military and nuclear power systems increased.…”

Verification & Validation in Medi SPICE

“…Verification and Validation (V&V) activities are important activities in the software development lifecycle and consume up to 50% of project development time [1], [2] and up to 50% of the total cost [3]. While both V&V play a key role in software development, there is a level of ambiguity in the use of both of these terms.…”