Searching under the Streetlight for Useful Software Analytics

IIEEE Trans. Software Eng.

Gousios

Panichella

et al. 2019

Software testing is one of the key activities to achieve software quality in practice. Despite its importance, however, we have a remarkable lack of knowledge on how developers test in real-world projects. In this paper, we report on a large-scale field study with 2,443 software engineers whose development activities we closely monitored over 2.5 years in four integrated development environments (IDEs). Our findings, which largely generalized across the studied IDEs and programming languages Java and C#, question several commonly shared assumptions and beliefs about developer testing: half of the developers in our study do not test; developers rarely run their tests in the IDE; most programming sessions end without any test execution; only once they start testing, do developers do it extensively; a quarter of test cases is responsible for three quarters of all test failures; 12% of tests show flaky behavior; Test-Driven Development (TDD) is not widely practiced; and software developers only spend a quarter of their time engineering tests, whereas they think they test half of their time. We compile these practices of loosely guiding one's development efforts with the help of testing in an initial summary on Test-Guided Development (TGD), a behavior we argue to be closer to the development reality of most developers than TDD.

Section: Related Tools and Pluginsmentioning

confidence: 99%

Developer Testing in the IDE: Patterns, Beliefs, and Behavior

IIEEE Trans. Software Eng.

Gousios

Panichella

et al. 2019

“…While several WATCHDOG-like plugins for IDE-instrumentation exist [42][43][44][45], none of them have been used to study the debugging behavior of developers, manifesting our knowledge gap of empirical debugging. Ko and Myers showed the practical usefulness of the "live IDE" wish expressed in SQ13 with their Whyline prototype [19], which helps developers reason about assumed program behavior.…”

Section: Related Workmentioning

confidence: 99%

On the dichotomy of debugging behavior among programmers

Proceedings of the 40th International Conference on Software Engineering

Spruit

Spinellis

et al. 2018

Debugging is an inevitable activity in most software projects, often difficult and more time-consuming than expected, giving it the nickname the "dirty little secret of computer science." Surprisingly, we have little knowledge on how software engineers debug software problems in the real world, whether they use dedicated debugging tools, and how knowledgeable they are about debugging. This study aims to shed light on these aspects by following a mixed-methods research approach. We conduct an online survey capturing how 176 developers reflect on debugging. We augment this subjective survey data with objective observations on how 458 developers use the debugger included in their integrated development environments (IDEs) by instrumenting the popular ECLIPSE and INTELLIJ IDEs with the purpose-built plugin WATCHDOG 2.0. To clarify the insights and discrepancies observed in the previous steps, we followed up by conducting interviews with debugging experts and regular debugging users. Our results indicate that IDE-provided debuggers are not used as often as expected, because "printf debugging" remains a feasible choice for many programmers. Furthermore, both knowledge and use of advanced debugging features are low. Our results call for strengthening hands-on debugging experience in computer science curricula and have already refined the implementation of modern IDE debuggers.

“…Jenny finds that, for her small change, she was well above average regarding TDD use: She learned how to develop TDD-style from the "Let's Developer" Youtube channel. 8 The WatchDog project for "Let's Developer" is the second highest TDD follower of all WatchDog users on 19th November, 2015 (following TDD for 40% of all modifications). 9 In TDD, programmers systematically coevolve production and test code, while constantly cycling between a state of succeeding and failing test cases.…”

Section: Summary Of Your Test-driven Development Practicesmentioning

confidence: 99%

“…WatchDog has an intermediate position, as it does both and also allows its users to make comparison among themselves. Hackystat with its Zorro extension was one of the first solutions that aimed at detecting TDD activities [7,8], similar to the education-oriented TDD-Guide [9] and the prototype TestFirstGauge [10]. In contrast to WatchDog, Hackystat did not focus on the IDE, but offered a multitude of sensors, from bug trackers like Bugzilla to build tools like ant.…”

Section: Related Toolsmentioning

confidence: 99%

How to catch 'em all

Proceedings of the 3rd International Workshop on Software Engineering Research and Industrial Practice

Levaja

Panichella

et al. 2016

As software engineering researchers, we are also zealous tool smiths. Building a research prototype is often a daunting task, let alone building an industry-grade family of tools supporting multiple platforms to ensure the generalizability of results. In this paper, we give advice to academic and industrial tool smiths on how to design and build an easy-to-maintain architecture capable of supporting multiple integrated development environments (IDEs). Our experiences stem from WatchDog, a multi-IDE infrastructure that assesses developer testing activities in vivo and that over 2,000 registered developers use. To these software engineering practitioners, WatchDog provides real-time and aggregated feedback in the form of individual testing reports.