An analysis of the relationship between conditional entropy and failed error propagation in software testing

Abstract

Failed error propagation (FEP) is known to hamper software testing, yet it remains poorly understood. We introduce an information theoretic formulation of FEP that is based on measures of conditional entropy. This formulation considers the situation in which we are interested in the potential for an incorrect program state at statement s to fail to propagate to incorrect output. We define five metrics that differ in two ways: whether we only consider parts of the program that can be reached after executing s and whether we restrict attention to a single program path of interest. We give the results of experiments in which it was found that on average one in 10 tests suffered from FEP, earlier studies having shown that this figure can vary significantly between programs. The experiments also showed that our metrics are well-correlated with FEP. Our empirical study involved 30 programs, for which we executed a total of 7,140,000 test cases. The results reveal that the metrics differ in their performance but the Spearman rank correlation with failed error propagation is close to 0.95 for two of the metrics. These strong correlations in an experimental setting, in which all information about both FEP and conditional entropy is known, open up the possibility in the longer term of devising inexpensive information theory based metrics that allow us to minimise the effect of FEP.

An Empirical Study on Mutation, Statement and Branch Coverage Fault Revelation That Avoids the Unreliable Clean Program Assumption

“…A mutant is strongly killed [9], [31], [32] if the original program and the mutant exhibit some observable difference in their output behaviour. Strong mutation does not subsume weak mutation because of potential failed error propagation [9], [33], which may cause state differences to be over-written by subsequent computation.…”

Effective fault localization of automotive Simulink models: achieving the trade-off between test oracle effort and fault localization accuracy

“…A recent study by Masri et al [48] shows the prevalence of CCT in test suites applied to programs for the purpose of testing or statistical debugging. Some recent papers [6,19] define information theoretic metrics to predict when an error at an intermediate program statement may fail to propagate to final outputs, and hence, potentially lead to CCT. Such metrics might help us improve fault localization accuracy by identifying and excluding test cases that are likely to be CCT when executed on the underlying Simulink model.…”

“…(5) We present the tool we have implemented to support different parts of our approach. (6) Finally, compared to our previous paper, we have expanded the related work and threats to validity discussions. This paper is organized as follows: In Section 2, we provide some background on Simulink models and fix our notation.…”

A mapping study on testing non-testable systems

“…Thus, research that explores methods of reducing the impact of coincidental correctness on these techniques would be valuable. For example, Clark and Hierons (2012) and Androutsopoulos et al (2014) developed a series of metrics that estimate the probability of encountering coincidental correctness on particular program paths. Such metrics can be used to select test cases that are less susceptible to coincidental correctness.…”