Test-Driven Code Review: An Empirical Study

Abstract

Test-Driven Code Review (TDR) is a code review practice in which a reviewer inspects a patch by examining the changed test code before the changed production code. Although this practice has been mentioned positively by practitioners in informal literature and interviews, there is no systematic knowledge of its effects, prevalence, problems, and advantages. In this paper, we aim at empirically understanding whether this practice has an effect on code review effectiveness and how developers' perceive TDR. We conduct (i) a controlled experiment with 93 developers that perform more than 150 reviews, and (ii) 9 semi-structured interviews and a survey with 103 respondents to gather information on how TDR is perceived. Key results from the experiment show that developers adopting TDR find the same proportion of defects in production code, but more in test code, at the expenses of fewer maintainability issues in production code. Furthermore, we found that most developers prefer to review production code as they deem it more critical and tests should follow from it. Moreover, general poor test code quality and no tool support hinder the adoption of TDR. Public preprint: [

Understanding flaky tests: the developer’s perspective

“…Our participants revealed that finding how to design a test code to avoid flakiness is an important challenge to face. This motivates the growing research area around test code quality [18,29,[33][34][35][36][37] and provides two promising directions that the research community can focus on: (i) the definition of a set of design patterns that can support the creation of deterministic tests; (ii) the definition of a set of flakiness-related anti-patterns that practitioners should avoid when writing test cases. While some initial steps have been done about the relation between test smells and flaky tests [31,32], further investigation is necessary.…”

Pizza versus Pinsa: On the Perception and Measurability of Unit Test Code Quality

“…Test code quality represents a multi-faceted concept able to express how useful a test will be for developers during the understanding of the production code [18], the debugging activities [19,20], and the early catching of defects [21]. Over the last decade, a number of researchers have been studying test code quality with the aim of defining metrics able to characterize it under different perspectives.…”

Primers or reminders?

“…However, authors could not find substantial evidence on the influence of change part ordering on mental load or review performance. Spadini et al [46] designed and conducted a controlled experiment to investigate whether examining changed test code before the changed production code (also known as Test Driven Code Review or TDR) affects code review effectiveness. According to the findings of Spadini et al, developers adopting TDR find the same amount of defects in production code, but more defects in test code and fewer maintainability issues in the production code.…”

“…We consider the other variables as control variables, which also include the time spent on the review, the participant's role, years of experience in Java and Code Review, and tiredness. Finally, we run a logistic regression model similar to the one used by McIntoshet al [28] and Spadini et al [46]. To ensure that the selected logistic regression model is appropriate for the available data, we first (1) compute the Variance Inflation Factors (VIF) as a standard test for multicollinearity, finding all the values to be below 3 (values should be below 10), thus indicating little or no multicollinearity among the independent variables, (2) run a multilevel regression model to check whether there is a significant variance among reviewers, but we found little to none, thus indicating that a single level regression model is appropriate, and, finally, (4) when building the model we added the independent variables step-by-step and found that the coefficients remained stable, thus further indicating little to no interference among the variables.…”

“…The remaining constructs we use are defined in previous publications, and we reuse the existing instruments as much as possible. For instance, the tool employed for the online experiment is based on similar tools used in earlier works [9,46].…”