Understanding user understanding: determining correctness of generated program invariants

Abstract

Recently, work has begun on automating the generation of test oracles, which are necessary to fully automate the testing process. One approach to such automation involves dynamic invariant generation, which extracts invariants from program executions. To use such invariants as test oracles, however, it is necessary to distinguish correct from incorrect invariants, a process that currently requires human intervention. In this work we examine this process. In particular, we examine the ability of 30 users, across two empirical studies, to classify invariants generated from three Java programs. Our results indicate that users struggle to classify generated invariants: on average, they misclassify 9.1% to 31.7% of correct invariants and 26.1%-58.6% of incorrect invariants. These results contradict prior studies that suggest that classification by users is easy, and indicate that further work needs to be done to bridge the gap between the effectiveness of dynamic invariant generation in theory, and the ability of users to apply it in practice. Along these lines, we suggest several areas for future work.

Does Automated Unit Test Generation Really Help Software Testers? A Controlled Empirical Study

“…These approaches typically differ in how the test oracle is constructed, and include: automatically inferring program invariants [Staats et al 2012b;Wei et al 2011]; automatically inferring parametrized test input assertions ; automatically producing concrete test input assertions (as is done by EVOSUITE) [Fraser and Arcuri 2011;Staats et al 2012a]. Currently, there is no scientific consensus which approach is preferable, and in practice all of these approaches appear to be used infrequently by industry.…”

Dodona: automated oracle data set selection

“…This is typical when evaluating testing approaches; early work refines the approach, after which human studies begin to rigorously assess the human factor (e.g. faultlocalization [16], invariant generation [19]). …”

“…In some cases, these approaches include methods for creating test oracles, but such approaches always -albeit often implicitly -require manual intervention by test engineers to inspect and correct the results [9,7]. Evidence supporting the effectiveness of these approaches is mixed, with user studies noting a tendency for test engineers to accept incorrect oracles [19,8].…”

“…Both bodies of work assume the tester will later manually correct generated test oracles, and are part of the "generate-and-fix" paradigm for test oracle construction. Work evaluating this paradigm with users is mixed, but on the whole discouraging [8,19].…”

“…While several approaches for automatically generating such oracles have been proposed, most of these approaches attempt to completely automate the process. This results in a "generate and fix" approach, whereby the generation process produces effective test oracles, but only if developers can correct the output from the tools, a challenging task [8,19]. In contrast, in this work, we seek not to completely automate oracle generation, but instead to support test engineers in the construction of expected value test oracles.…”