Tree-Based Methods for Classifying Software Failures

Francis, Patrick; León, David De; Minch, M.; Podgurski, Andy

doi:10.1109/issre.2004.43

Cited by 66 publications

(52 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Chan et al [5] further investigated the feasibility of using pattern classification techniques when the test outputs cannot be accurately determined. Podgurski and colleagues [32][33] classified failure reports into categories via classifiers, and then refined the classification with the aim to extract more information to help testers diagnose program failure s. Bowring et al [34] used a progressive approach to train a classifier to ease the test oracle problem in regression testing. Chan et al [35] used classifiers to identify different types of behaviors related to the synchronization failures of objects in a multimedia application.…”

Section: Related Workmentioning

confidence: 99%

Experimental Study to Compare the Use of Metamorphic Testing and Assertion Checking

Zhang¹,

Chan²,

Tse³

et al. 2009

Journal of Software

View full text Add to dashboard Cite

A test oracle in software testing is a mechanism for checking whether the program under test behaves correctly for any execution. In some practical situations, oracles can be unavailable or too expensive to apply.Metamorphic testing (MT) was proposed to alleviate this problem so that software can be delivered under the time-to-market pressure. However, the effectiveness of MT has not been studied adequately. This paper conducts a controlled experiment to investigate the cost effectiveness of using MT. The fault detection capability and time cost of MT are compared with the standard assertion checking method. Our results show that MT has potentials to detect more faults than the assertion checking method. The experimental results also show a trade-off between the two testing methods: MT can be less efficient but more effective, and can be defined at a coarser level of granularity th an the assertion checking method.

show abstract

Section: Related Workmentioning

confidence: 99%

Experimental Study to Compare the Use of Metamorphic Testing and Assertion Checking

Zhang¹,

Chan²,

Tse³

et al. 2009

Journal of Software

View full text Add to dashboard Cite

show abstract

“…On the other hand, if our goal were simply to find a single test case triggering each fault, an obvious approach would be to cluster the test cases and then select a single test from each cluster, as in previous approaches to the problem [9,28]. The FPF algorithm we use is itself based on the idea of approximating optimal clusters [10]; we simply ignore the clustering aspect and use only the ranking information.…”

Section: Clustering As An Alternative To Furthest Point Firstmentioning

confidence: 99%

“…This produced better results than our single-vector method, and it was also more efficient, as it did not require the use of large vectors combining multiple features. This approach is essentially a completely unsupervised variation (with the addition of some recent advances in clustering) of earlier approaches to clustering test cases that trigger the same bug [9]. Our approach is unsupervised because we exploit testcase reduction as a way to select relevant features, rather than relying on the previous approaches' assumption that features useful in predicting failure or success would also distinguish failures from each other.…”

Section: Clustering As An Alternative To Furthest Point Firstmentioning

confidence: 99%

Taming compiler fuzzers

Yang

Groce

Zhang

et al. 2013

Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation

120

View full text Add to dashboard Cite

Aggressive random testing tools ("fuzzers") are impressively effective at finding compiler bugs. For example, a single test-case generator has resulted in more than 1,700 bugs reported for a single JavaScript engine. However, fuzzers can be frustrating to use: they indiscriminately and repeatedly find bugs that may not be severe enough to fix right away. Currently, users filter out undesirable test cases using ad hoc methods such as disallowing problematic features in tests and grepping test results. This paper formulates and addresses the fuzzer taming problem: given a potentially large number of random test cases that trigger failures, order them such that diverse, interesting test cases are highly ranked. Our evaluation shows our ability to solve the fuzzer taming problem for 3,799 test cases triggering 46 bugs in a C compiler and 2,603 test cases triggering 28 bugs in a JavaScript engine.

show abstract

“…Engineering It is a relatively new topic to apply machine learning to software quality engineering [4,14,18,23,42]. Brun and Ernst [4] isolated fault-revealing properties of a program by applying machine learning to a faulty program and its fixed version.…”

Section: Application Of Machine Learning In Software Qualitymentioning

confidence: 99%

“…Then the fault-revealing properties are used to identify other potential faults. Francis et al [14] proposed two new tree-based techniques to classify failing executions so that the failing executions resulting from the same cause are grouped together. Podgurski et al [42] proposed to use supervised and unsupervised pattern classification and multivariate visualization to classify failing executions with the related cause.…”

Section: Application Of Machine Learning In Software Qualitymentioning

confidence: 99%

Is This a Bug or an Obsolete Test?

Hao

Lan

Zhang

et al. 2013

ECOOP 2013 – Object-Oriented Programming

View full text Add to dashboard Cite

Abstract. In software evolution, developers typically need to identify whether the failure of a test is due to a bug in the source code under test or the obsoleteness of the test code when they execute a test suite. Only after finding the cause of a failure can developers determine whether to fix the bug or repair the obsolete test. Researchers have proposed several techniques to automate test repair. However, test-repair techniques typically assume that test failures are always due to obsolete tests. Thus, such techniques may not be applicable in real world software evolution when developers do not know whether the failure is due to a bug or an obsolete test. To know whether the cause of a test failure lies in the source code under test or in the test code, we view this problem as a classification problem and propose an automatic approach based on machine learning. Specifically, we target Java software using the JUnit testing framework and collect a set of features that may be related to failures of tests. Using this set of features, we adopt the Best-first Decision Tree Learning algorithm to train a classifier with some existing regression test failures as training instances. Then, we use the classifier to classify future failed tests. Furthermore, we evaluated our approach using two Java programs in three scenarios (within the same version, within different versions of a program, and between different programs), and found that our approach can effectively classify the causes of failed tests.

show abstract

Tree-Based Methods for Classifying Software Failures

Cited by 66 publications

References 17 publications

Experimental Study to Compare the Use of Metamorphic Testing and Assertion Checking

Experimental Study to Compare the Use of Metamorphic Testing and Assertion Checking

Taming compiler fuzzers

Is This a Bug or an Obsolete Test?

Contact Info

Product

Resources

About