Validation of Software Testing Experiments: A Meta-Analysis of ICST 2013

Abstract

Abstract-Researchers in software testing are often faced with the following problem of empirical validation: does a new testing technique actually help analysts find more faults than some baseline method? Researchers evaluate their contribution using statistics to refute the null hypothesis that their technique is no better at finding faults than the state of the art. The decision as to which statistical methods are appropriate is best left to an expert statistician, but the reality is that software testing researchers often don't have this luxury. We developed an algorithm, MeansTest, to help automate some aspects of statistical analysis. We implemented MeansTest in the statistical software environment R, encouraging reuse and decreasing the need to write and test statistical analysis code. Our experiment showed that MeansTest has significantly higher F-measures than several other common hypothesis tests. We applied MeansTest to systematically validate the work presented at the 2013 IEEE Sixth International Conference on Software Testing, Verification, and Validation (ICST'13). We found six papers that potentially misstated the significance of their results. MeansTest provides a free and easy-to-use possibility for researchers to check whether their chosen statistical methods and the results obtained are plausible. It is available for download at coest.org.

DeepCNN: A Dual Approach to Fault Localization and Repair in Convolutional Neural Networks

“…In recent years, researchers have introduced numerous fault localization methods to pinpoint the root causes of failures in traditional software programs. These methods fall into eight main categories [18]: spectrum-based [19], [20], slice-based [21], [22], statistics-based [23], [24], program state-based [25], machine learning-based [26], [27], modelbased [28], data mining-based [29], and miscellaneous techniques [30]. Numerous studies have been undertaken to assess the efficiency of these fault localization techniques.…”

A Survey of Challenges in Spectrum-Based Software Fault Localization

“…Then, they use statistics to demonstrate that a proposed new technique locates faults ''significantly'' better than the state of the art. As they are not statisticians and do not have statisticians readily available, this may lead to incorrect statistical analysis and conclusions about the importance of their SBFL results [38].…”

“…Besides, statistical tools are needed to help developers evaluate their results. For example, the authors in [38], [39] presented the first such tools called ''MeansTest''. The tool automates some aspects of the statistical analysis of results by checking whether the statistical methods used and the results obtained are both plausible.…”

Methods for Enhancing Software Fault Localization

“…In other words, they use statistics to demonstrate that a proposed new technique locates faults "significantly" better than the state-of-the-art. As they are not statisticians and do not have statisticians readily available, this may lead to incorrect statistical analysis and conclusions about the importance of their SBFL results [54].…”

“…Figure 3.1 shows the paper selection process and its outcome at each stage. In addition, all the papers obtained after applying the paper selection process are listed below: [168], [132], [6], [155], [34], [43], [176], [175], [5], [54], [55], [1], [69], [129], [18], [4], [142], [183], [11], [36], [156], [165], [174], [92], [141], [98], [99], [106], [127], [67], [25], [76], [49], [61], [20], [143], [23], [118], [10], [41], [38], [139], [84], [79], [104], [112], [85], [9]<...…”