The impact of code review coverage and code review participation on software quality: a case study of the qt, VTK, and ITK projects

Abstract

Software code review, i.e., the practice of having third-party team members critique changes to a software system, is a well-established best practice in both open source and proprietary software domains. Prior work has shown that the formal code inspections of the past tend to improve the quality of software delivered by students and small teams. However, the formal code inspection process mandates strict review criteria (e.g., in-person meetings and reviewer checklists) to ensure a base level of review quality, while the modern, lightweight code reviewing process does not. Although recent work explores the modern code review process qualitatively, little research quantitatively explores the relationship between properties of the modern code review process and software quality. Hence, in this paper, we study the relationship between software quality and: (1) code review coverage, i.e., the proportion of changes that have been code reviewed, and (2) code review participation, i.e., the degree of reviewer involvement in the code review process. Through a case study of the Qt, VTK, and ITK projects, we find that both code review coverage and participation share a significant link with software quality. Low code review coverage and participation are estimated to produce components with up to two and five additional post-release defects respectively. Our results empirically confirm the intuition that poorly reviewed code has a negative impact on software quality in large systems using modern reviewing tools.

ReDA: A Web-Based Visualization Tool for Analyzing Modern Code Review Dataset

“…We found that from 374 reviews 6 created on 23/12/2011, there are 356 reviews (98%) that were abandoned on 29/12/2010. Moreover, these reviews were created by the same developer at consecutive times.…”

“…Extracting knowledge from these datasets has produced promising research with the goal of improving the software quality and software development process. Recently, many studies have used code review datasets to understand and improve both review effort [2]- [5] and review quality [6], [7]. However, a raw code review dataset is generally imperfect since the data collection process in each support tool 1 https://code.google.com/p/gerrit/ 2 https://www.reviewboard.org/ 3 https://code.google.com/p/rietveld/ varies in methodology, accuracy, and degree of automation [8].…”

Towards quality gates in continuous delivery and deployment

“…Stewart et al [12] tried to relate code quality to factors such as release frequency, number of releases and the changes in size of the code base across releases in open source software projects, but could not derive any conclusions. McIntosh et al [13] studied the impact of code review coverage on software quality. Baccelli and Bird [14] investigated the outcomes and challenges of modern tool-based code reviews and observed that, amongst other things, code reviews are not only used for revealing defects, but also serve to promote awareness and knowledge transfer.…”

A case study on machine learning model for code review expert system in software engineering

“…al. [4] summarized their case study with statement: "Components with higher review coverage tend to have fewer post-release defects". Their analysis also indicates that "Although review coverage is negatively associated with software quality in our models, several defect-prone components have high coverage rates, suggesting that other properties of the code review process are at play."…”