RANDOM VS. SCENARIO-BASED VS. FAULT-BASED TESTING - An Industrial Evaluation of Formal Black-Box Testing Methods

Abstract

Given a formal model of a system under test there are different strategies for deriving test cases from such a model systematically. These strategies are based on different underlying testing objectives and concepts. Obviously, their usage has impact on the generated test cases. In this paper we evaluate random, scenariobased and fault-based test case generation strategies in the context of an industrial application and assess the advantages and disadvantages of these three strategies. The derived test cases are evaluated in terms of coverage and in terms of the detected errors on a commercial and on an open source implementation of the Voice-Over-IP Session Initiation Protocol.