Empirical study of correlation between mutation score and model inference based test suite adequacy assessment

Abstract

In this paper we investigate a method for test suite evaluation that is based on an inferred model from the test suite. The idea is to use the similarity between the inferred model and the system under test as a measure of test suite adequacy, which is the ability of a test suite to expose errors in the system under test. We define similarity using the root mean squared error computed from the differences of the system under test output and the model output for certain inputs not used for model inference. In the paper we introduce the approach and provide results of an experimental evaluation where we compare the similarity with the mutation score. We used the Pearson Correlation coefficient to calculate whether a linear correlation between mutation score and root mean squared error exists. As a result we obtain that in certain cases the computed similarity strongly correlates with the mutation score.

Mutation Score, Coverage, Model Inference: Quality Assessment for T-Way Combinatorial Test-Suites

“…of t = 6 is enough to detect all faults. The authors of [17] provide an empirical study on the correlation of test-suite effectiveness between mutation score and model inference based test-suite quality assessment. Their results show that test-suite quality assessment without executing the program is possible, where the results correlate with a mutation score of the test-suite, but the applicability depends on the inputs and structural properties of the program under test.…”

“…The decision tree learning method we used in this work is the C4.5 algorithm as introduced in [6]. Other than [23] and [17] where model inference is used to evaluate the effectiveness or quality of a test-suite by comparing the test-suite to the program under test, we compute the quality valuation by comparing test-suites by each other. The test-suite for which we assess its quality is compared to the test-suite TS tmax .…”

Test-Suite Reduction Does Not Necessarily Require Executing the Program under Test

“…3) Triangle: A function that determines either one of the triangle types: equilateral, scalene, or isosceles, for valid inputs, otherwise it returns: no triangle [23]. 4) POP3: An implementation of the state machine in [17]. 5) Car Alarm System (CAS): An implementation of the state machine in [16].…”

“…The authors demonstrate that the reduced test-suite retains all of the fault finding capability of the original test-suite by using mutation testing, which also holds for our approach. In [17] the authors provide an empirical evaluation of the correlation between mutation score and a model inference based test-suite adequacy assessment method. Briand et al [15] describe a testsuite refinement approach that relies on the black box testing technique Category Partition and machine learning.…”

“…The empirical evaluation is based on 7 example programs. These examples and examples of similar size (size in lines of code) were already used to evaluate model learning, test-suite reduction, and test case generation approaches (e.g., [13], [14], [15], [16], [17]). The obtained results of our model learning based reduction approach are very promising, showing substantial reductions and almost no decline of coverage and mutation score.…”