Test Set Diameter: Quantifying the Diversity of Sets of Test Cases

Abstract

Abstract-A common and natural intuition among software testers is that test cases need to differ if a software system is to be tested properly and its quality ensured. Consequently, much research has gone into formulating distance measures for how test cases, their inputs and/or their outputs differ. However, common to these proposals is that they are data type specific and/or calculate the diversity only between pairs of test inputs, traces or outputs.We propose a new metric to measure the diversity of sets of tests: the test set diameter (TSDm). It extends our earlier, pairwise test diversity metrics based on recent advances in information theory regarding the calculation of the normalized compression distance (NCD) for multisets. An advantage is that TSDm can be applied regardless of data type and on any test-related information, not only the test inputs. A downside is the increased computational time compared to competing approaches.Our experiments on four different systems show that the test set diameter can help select test sets with higher structural and fault coverage than random selection even when only applied to test inputs. This can enable early test design and selection, prior to even having a software system to test, and complement other types of test automation and analysis. We argue that this quantification of test set diversity creates a number of opportunities to better understand software quality and provides practical ways to increase it.

Tester interactivity makes a difference in search-based software testing: A controlled experiment

“…We cannot state, in general, that this is a good measure for the number of faults found by the test suite. However, there is evidence to suggest that increase diversity does result in more fault-revealing test suites [23,24].…”

“…Only that such a tool increases the number of SUT behaviors that are being investigated. There is, however, work to suggest that test case diversity does result in greater structural and fault coverage [24]. Further studies are necessary before any conclusions can be drawn regarding the impact of such methods on software quality in general and on fault finding in particular.…”

“…Work by Feldt et al links test suites with higher diversity to higher structural and fault coverage [23,24].…”

Similarity-based prioritization of test case automation

“…Research effort have also been devoted to similarity-based selection techniques, in particular, in the model-based domain (Hemmati et al 2013). Another class of related papers represents the opposite direction of test similarity research and measures the diversity between two test sets (Shi et al 2016;Feldt et al 2015). These techniques may be used as an inverse indicator of test case similarity.…”

Empirical evaluation of mutation‐based test case prioritization techniques

“…Recent advances in test case prioritization focus on identifying and promoting the diversity of the selected test cases [], rather than maximizing the coverage. This trend can provide several advantages, especially in cases where there is no source code availability [].…”