An empirical study about the effectiveness of debugging when random test cases are used

Abstract

Abstract-Automatically generated test cases are usually evaluated in terms of their fault revealing or coverage capability. Beside these two aspects, test cases are also the major source of information for fault localization and fixing. The impact of automatically generated test cases on the debugging activity, compared to the use of manually written test cases, has never been studied before.In this paper we report the results obtained from two controlled experiments with human subjects performing debugging tasks using automatically generated or manually written test cases. We investigate whether the features of the former type of test cases, which make them less readable and understandable (e.g., unclear test scenarios, meaningless identifiers), have an impact on accuracy and efficiency of debugging. The empirical study is aimed at investigating whether, despite the lack of readability in automatically generated test cases, subjects can still take advantage of them during debugging.

Automatically generated patches as debugging aids: a human study

“…Research has also identified other reasons why developers are not comfortable with blindly trusting auto-generated code. For example, generated code might be less readable and maintainable [7] [13].…”

Recent Advances in Automatic Black-Box Testing

“…However, a recent study by Ceccato et al revealed that random test cases are not harder to debug than manual test cases, on the contrary, due to their intrinsic simplicity, are often easier to debug [28].…”

“…Pure RT [14][15][16][17][18][19] Adaptive RT [20][21][22][23][24] Recent advances in random testing (2010)(2011)(2012)(2013)(2014) Guided RT [25][26][27][28] Fuzz testing [29][30][31] …”

Taming compiler fuzzers

“…This has been largely solved by Delta Debugging [45], an automated greedy search for small failure-inducing test cases. In fact, there is some evidence that debugging based on short random test cases is easier than debugging based on human-created test cases [3]. A second problem is the sheer volume of output: an overnight run of a fuzzer may result in hundreds or thousands of failure-inducing test cases.…”