Achieving scalable model-based testing through test case diversity

Abstract

The increase in size and complexity of modern software systems requires scalable, systematic, and automated testing approaches. Model-based testing (MBT), as a systematic and automated test case generation technique, is being successfully applied to verify industrial-scale systems and is supported by commercial tools. However, scalability is still an open issue for large systems, as in practice there are limits to the amount of testing that can be performed in industrial contexts. Even with standard coverage criteria, the resulting test suites generated by MBT techniques can be very large and expensive to execute, especially for system level testing on real deployment platforms and network facilities. Therefore, a scalable MBT technique should be flexible regarding the size of the generated test suites and should be easily accommodated to fit resource and time constraints. Our approach is to select a subset of the generated test suite in such a way that it can be realistically executed and analyzed within the time and resource constraints, while preserving the fault revealing power of the original test suite to a maximum extent. In this article, to address this problem, we introduce a family of similarity-based test case selection techniques for test suites generated from state machines. We evaluate 320 different similarity-based selection techniques and then compare the effectiveness of the best similarity-based selection technique with other common selection techniques in the literature. The results based on two industrial case studies, in the domain of embedded systems, show significant benefits and a large improvement in performance when using a similarity-based approach. We complement these analyses with further studies on the scalability of the technique and the effects of failure rate on its effectiveness. We also propose a method to identify optimal tradeoffs between the number of test cases to run and fault detection.

On the Correlation between the Effectiveness of Metamorphic Relations and Dissimilarities of Test Case Executions

“…In software testing literature there has been much work on test case selection and prioritization techniques using the concept of "similarity" of test cases [21]. The present research is different from these techniques as our objective is to reveal the nature of good MRs, rather than good individual test cases.…”

Similarity-based regression test case prioritization

“…Test cases were ordered before executing them. Similarly, Hemmati et al introduced a family of similarity-based test case selection techniques for model-based testing [13], [15]. The above two methods were applied to black-box testing.…”

“…Similarity-based test case prioritization techniques assume that test case diversity aids to detect more faults [15], [13]. Clustering-based prioritization techniques assume that the test cases within the same cluster have the same fault detection capability.…”

Evaluating and Comparing Fault-Based Testing Strategies for General Boolean Specifications: A Series of Experiments

“…The basic intuition of these extension guidelines is that the test cases should have certain degree of diversity. The concept of diversity has been widely used in other test case selection methods [9,5,21]. The details of the MUMCUT extensions proposed in our previous work are presented as follows.…”