Improving Development Practices through Experimentation: An Industrial TDD Case

Abstract

Test-Driven Development (TDD), an agile development approach that enforces the construction of software systems by means of successive micro-iterative testing coding cycles, has been widely claimed to increase external software quality. In view of this, some managers at Paf-a Nordic gaming entertainment company-were interested in knowing how would TDD perform at their premises. Eventually, if TDD outperformed their traditional way of coding (i.e., YW, short for Your Way), it would be possible to switch to TDD considering the empirical evidence achieved at the company level. We conduct an experiment at Paf to evaluate the performance of TDD, YW and the reverse approach of TDD (i.e., ITL, short for Iterative-Test Last) on external quality. TDD outperforms YW and ITL at Paf. Despite the encouraging results, we cannot recommend Paf to immediately adopt TDD as the difference in performance between YW and TDD is small. However, as TDD looks promising at Paf, we suggest to move some developers to TDD and to run a future experiment to compare the performance of TDD and YW. TDD slightly outperforms ITL in controlled experiments for TDD novices. However, more industrial experiments are still needed to evaluate the performance of TDD in real-life contexts.

Why Research on Test-Driven Development is Inconclusive?

“…[32] Impact of test-driven development on productivity, code and tests: A controlled experiment Fucci et al [14] A Dissection of the Test-Driven Development Process: Does It Really Matter to Test-First or to Test-Last? Dogša and Batic [10] The effectiveness of test-driven development : an industrial case study Fucci and Turhan [16] A Replicated Experiment on the Effectiveness of Test-first Development Thomson et al [44] What Makes Testing Work: Nine Case Studies of Software Development Teams Romano et al [35] Findings from a multi-method study on test-driven development Buchan et al [7] Causal Factors, Benefits and Challenges of Test-Driven Development: Practitioner Perceptions Scanniello et al [38] Students' and Professionals' Perceptions of Test-driven Development: A Focus Group Study Beller et al [4] Developer Testing in The IDE: Patterns, Beliefs, And Behavior Bannerman and Martin [1] A multiple comparative study of test-with development product changes and their effects on team speed and product quality Table 3: The primary studies collected in the third step Authors Title Karac et al [23] A Controlled Experiment with Novice Developers on the Impact of Task Description Granularity on Software Quality in Test-Driven Development Tosun et al [46] Investigating the Impact of Development Task on External Quality in Test-Driven Development: An Industry Experiment Borle et al [6] Analyzing the effects of test driven development in GitHub Fucci et al [15] A longitudinal cohort study on the retainment of test-driven development Kazerouni et al [24] Assessing Incremental Testing Practices and Their Impact on Project Outcomes Santos et al [37] Improving Development Practices through Experimentation : an Industrial TDD Case Tosun et al [45] On the Effectiveness of Unit Tests in Test-driven Development Santos et al [36] Does the Performance of TDD Hold Across Software Companies and Premises? A Group of Industrial Experiments on TDD Romano et al [34] An Empirical Assessment on Affective Reactions of Novice Developers When Applying Test-Driven Development Sundelin et al [43] Test-Driving FinTech Product Development: An Experience Report In terms of test quality, research has explored the quality of tests by measuring mutation scores (i.e., the bug detection ability of the tests) and code coverage (i.e., the degree to which the source code of a program is executed when a test suite runs).…”

“…Synthetic tasks vs. real-world tasksSynthetic taskRomano et al[35], Fucci and Turhan[16], Tosun et al[45], Pančur and Ciglaric[32], Karac et al[23], Tosun et al[46], Fucci et al[14], Santos et al[37], Fucci et al…”

“…What TDD is compared toIterative test lastTosun et al[45], Pančur and Ciglaric[32], Kazerouni et al[24], Fucci et al[14], Santos et al[37], Tosun et al[46] …”

“…task Romano et al[35], Fucci and Turhan[16], Tosun et al[45], Pančur and Ciglaric[32], Karac et al[23], Tosun et al[46], Fucci et al[14], Santos et al[37], Fucci et al[15], Kazerouni et al[24] Real task Thomson et al[44], Dogša and Batic[10] …”

Teaching C++ programming using automated unit testing and test‐driven development—Design and efficacy study

“…Among the various best practices of agile software development processes, automated unit testing (AUT) and testdriven development (TDD) have received significant attention in recent years [23,9,1,32,40,18]. Often, both of these practices are used together to reap maximum benefits.…”