Cause reduction: delta debugging, even without bugs

Groce, Alex; Alipour, Mohammad Amin; Zhang, Chaoqiang; Chen, Yang; Regehr, John

doi:10.1002/stvr.1574

Cited by 35 publications

(11 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This practice also helps to elevate the performance of symbolic execution [43]. Finally, each test 'case' is essentially a test input that needs a test oracle [44], in order to determine its corresponding output.…”

Section: Test Suites Usedmentioning

confidence: 99%

An Empirical Study on Mutation, Statement and Branch Coverage Fault Revelation That Avoids the Unreliable Clean Program Assumption

Chekam

Papadakis

Traon

et al. 2017

2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE)

117

116

View full text Add to dashboard Cite

Abstract-Many studies suggest using coverage concepts, such as branch coverage, as the starting point of testing, while others as the most prominent test quality indicator. Yet the relationship between coverage and fault-revelation remains unknown, yielding uncertainty and controversy. Most previous studies rely on the Clean Program Assumption, that a test suite will obtain similar coverage for both faulty and fixed ('clean') program versions. This assumption may appear intuitive, especially for bugs that denote small semantic deviations. However, we present evidence that the Clean Program Assumption does not always hold, thereby raising a critical threat to the validity of previous results. We then conducted a study using a robust experimental methodology that avoids this threat to validity, from which our primary finding is that strong mutation testing has the highest fault revelation of four widely-used criteria. Our findings also revealed that fault revelation starts to increase significantly only once relatively high levels of coverage are attained.

show abstract

Section: Test Suites Usedmentioning

confidence: 99%

An Empirical Study on Mutation, Statement and Branch Coverage Fault Revelation That Avoids the Unreliable Clean Program Assumption

Chekam

Papadakis

Traon

et al. 2017

2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE)

117

116

View full text Add to dashboard Cite

show abstract

“…While running larger (or more) test cases is desirable for thorough testing, time and resources limit the size (or number) of test cases that can be executed. Test-case minimization is used to minimize the testing cost in terms of execution time [6,39,40,94,[130][131][132]. The goal is to generate a smaller test case that has similar or (ideally) the same quality as the original test case; e.g., covers the same statements.…”

Section: Test-case Minimizationmentioning

confidence: 99%

“…We apply minimization to seven benchmarks and show that using [40]. Intuitively, Min can improve accuracy because it can be analyzed more comprehensively due to the improved analysis speed, whereas Ref can be prohibitively expensive to analyze in its entirety [49,117].…”

Section: Introductionmentioning

confidence: 99%

Minotaur

Mahmoud

Venkatagiri

Ahmed

et al. 2019

Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Syst

View full text Add to dashboard Cite

With the end of conventional CMOS scaling, efficient resiliency solutions are needed to address the increased likelihood of hardware errors. Silent data corruptions (SDCs) are especially harmful because they can create unacceptable output without the user's knowledge. Several resiliency analysis techniques have been proposed to identify SDC-causing instructions, but they remain too slow for practical use and/or sacrifice accuracy to improve analysis speed. We develop Minotaur, a novel toolkit to improve the speed and accuracy of resiliency analysis. The key insight behind Minotaur is that modern resiliency analysis has many conceptual similarities to software testing; therefore, adapting techniques from the rich software testing literature can lead to principled and significant improvements in resiliency analysis. Minotaur identifies and adapts four concepts from software testing: 1) it introduces the concept of input quality criteria for resiliency analysis and identifies PC coverage as a simple but effective criterion; 2) it creates (fast) minimized inputs from (slow) standard benchmark inputs, using the input quality criteria to assess the goodness of the created input; 3) it adapts the concept of test case prioritization to prioritize error injections and invoke early termination for a given instruction to speed up error-injection campaigns; and 4) it further adapts test case or input prioritization to accelerate SDC discovery across multiple inputs. We evaluate Minotaur by applying it to Approxilyzer, a state-of-the-art resiliency analysis tool. Minotaur's first three techniques speed up Approxilyzer's resiliency analysis by

show abstract

“…The output is a smaller program exhibiting this behavior. We will use the term "delta debugging" throughout this text, although a more appropriate name is the generalized concept of "cause reduction" [6]. The difference is that we are using the technique to identify causes of a wide range of analysis behaviors, and not just buggy behaviors.…”

Section: Delta Debuggingmentioning

confidence: 99%

Systematic approaches for increasing soundness and precision of static analyzers

Andreasen

Møller

Nielsen

2017

Proceedings of the 6th ACM SIGPLAN International Workshop on State of the Art in Program Analysis

View full text Add to dashboard Cite

Building static analyzers for modern programming languages is difficult. Often soundness is a requirement, perhaps with some well-defined exceptions, and precision must be adequate for producing useful results on realistic input programs. Formally proving such properties of a complex static analysis implementation is rarely an option in practice, which raises the challenge of how to identify causes and importance of soundness and precision problems. Through a series of examples, we present our experience with semi-automated methods based on delta debugging and dynamic analysis for increasing soundness and precision of a static analyzer for JavaScript. The individual methods are well known, but to our knowledge rarely used systematically and in combination.

show abstract

Cause reduction: delta debugging, even without bugs

Cited by 35 publications

References 65 publications

An Empirical Study on Mutation, Statement and Branch Coverage Fault Revelation That Avoids the Unreliable Clean Program Assumption

An Empirical Study on Mutation, Statement and Branch Coverage Fault Revelation That Avoids the Unreliable Clean Program Assumption

Minotaur

Systematic approaches for increasing soundness and precision of static analyzers

Contact Info

Product

Resources

About