Revealing influence of model structure and test case profile on the prioritization of test cases in the context of model-based testing

Abstract

Background: Test case prioritization techniques aim at defining an order of test cases that favor the achievement of a goal during test execution, such as revealing failures as earlier as possible. A number of techniques have already been proposed and investigated in the literature and experimental results have discussed whether a technique is more successful than others. However, in the context of model-based testing, only a few attempts have been made towards either proposing or experimenting test case prioritization techniques. Moreover, a number of factors that may influence on the results obtained still need to be investigated before more general conclusions can be reached. Methods: In order to evaluate factors that potentially affect the performance of test case prioritization techniques, we perform three empirical studies, an exploratory one and two experiments. The first study focus on expose the techniques to a common and fair environment, since the investigated techniques have never been studied together, and observe their behavior. The following two experiments aim at observing the effects of two factors: the structure of the model and the profile of the test cases that fail. We designed the experiments using the one-factor-at-a-time strategy. Results: The first study suggests that the investigated techniques performs differently, however other factors, aside from the test suites and number of failures, affect the techniques, motivating further investigation. As results from the two experiments, on one hand, the model structure do not affect significantly the investigated techniques. On the other hand, we are able to state that the profile of the test case that fails may have a definite influence on the performance of the techniques investigated. Conclusions: Through these studies, we conclude that, a fair evaluation involving test case prioritization techniques must take into account, in addition to the techniques and the test suites, different characteristics of the test cases that fail as variable.

Enhanced Adaptive Random Test Case Prioritization for Model-based Test Suites

“…Empirical study [24] performed by João Felipe Silva Ouriques et al investigates the effect of the model structure and characteristics of test cases that fail on the fault detection capability of several TCP techniques. Study results show that the characteristics of failed test cases affect the investigated techniques more significantly than the model layout.…”

Measuring the Efficiency of MFWA Technique for Prioritizing Event Sequences Test Cases

“…3.2 Redundancy [12] stated that a large number of test cases may caused redundancy. The sequence of an event test case can produce a large test size since it is the behaviour of the event sequence test case to have large input of sequences, which can sometimes be infinite.…”

A Hint-Based Technique for System Level Model-Based Test Case Prioritization

“…To the best of our knowledge, there is no other initiative of applying ARP in the MBT context. In previous work, we conducted a set of studies, as already discussed in Section 1 [21] including the two variants, with Jaccard [11] and Manhattan [22] distance functions. Besides our main results, in the referred context, Jaccard function performed better than the Manhattan.…”

“…In a previous work [21], we performed a set of empirical studies evaluating how several factors, such as number of faults, elements of model's layout (represented by the amount of branches, joins, and loops it has), and failing test cases' characteristics, can affect the behavior of traditional TCP techniques in the MBT context, focusing on Labeled Transition Systems (LTS). Our results showed that, for models with different layouts, the investigated techniques did not present variation in fault detection capability, which indicates low effects of model layout.…”