When, how, and why developers (do not) test in their IDEs

Abstract

The research community in Software Engineering and Software Testing in particular builds many of its contributions on a set of mutually shared expectations. Despite the fact that they form the basis of many publications as well as open-source and commercial testing applications, these common expectations and beliefs are rarely ever questioned. For example, Frederic Brooks' statement that testing takes half of the development time seems to have manifested itself within the community since he first made it in the "Mythical Man Month" in 1975. With this paper, we report on the surprising results of a large-scale field study with 416 software engineers whose development activity we closely monitored over the course of five months, resulting in over 13 years of recorded work time in their integrated development environments (IDEs). Our findings question several commonly shared assumptions and beliefs about testing and might be contributing factors to the observed bug proneness of software in practice: the majority of developers in our study does not test; developers rarely run their tests in the IDE; Test-Driven Development (TDD) is not widely practiced; and, last but not least, software developers only spend a quarter of their work time engineering tests, whereas they think they test half of their time.

How developers debug

“…5) The phenomenon of a discrepancy between survey answers and observed behavior is not new. We observed it similarly with developers claiming to test, and then not nearly testing as much in reality [37]. As a consequence, we emphasize our previous finding that survey answers always be cross-validated by other methods.…”

“…Our study results WO5 to WO7 point to the fact that debugging in most cases is a short, "getit-done" (I1) type of activity that, with only 14% of active IDE time (WO5) we found to consume significantly less than the 30 − 90% for testing and debugging reported by Beizer [42] and than the estimations by our interviewees, who gave a range of 20% to 60% of their active work time. One reason why our measured range is so much lower is that developers (and humans in general) seem to have a tendency to overestimate the duration of unpleasant tasks, as we previously observed with testing [37]. Another might be that developers included debugging tasks such as printf and the use of external tools, which we cannot measure.…”

“…To investigate the debugging habits of developers in the IDE, we extended our existing WATCHDOG [37]- [39] infrastructure to also track developers' debugging behavior for RQ2, resulting in WATCHDOG 2.0 for both ECLIPSE and INTELLIJ. The original WATCHDOG was used to verify common expectations and beliefs about testing [37], [38]. WATCHDOG is technically centered around the concept of intervals that capture the start and end of common development activities like reading and writing code as well as running JUNIT tests.…”

Usage of Tests in an Open-Source Community

“…We could observe whether there is the same phenomenon in an open-source community. Blondeau et al [2017] also took inspiration from Gligoric et al [2014] and Beller et al [2015]. The results of these two other papers will be compared to ours in Section 4.…”

“…Thanks to Pharo, we had all the necessary tools to detect users code changes and their execution of tests. So, we collected precise information like Beller et al [2015]. We collect test informa-tion through a plugin that were developed for Pharo.…”