The Impact of a Low Level of Agreement Among Reviewers in a Code Review Process

Abstract

Part 1: Full PapersInternational audienceSoftware code review systems are commonly used in software development. In these systems, many patches are submitted to improve the quality. To verify the quality, voting is commonly used by contributors; however, there still exists a major problem, namely, that reviewers do not always simply reach a broad agreement. In our previous study, we found that consensus is not usually reached, implying that an individual reviewer’s final decision usually differs from that of the majority of the other reviewers. In this study, we further investigate the reasons why such situations often occur, and provide suggestions for better handling of these problems. Our analysis of the Qt and OpenStack project datasets allow us to suggest that a patch owner should select more appropriate reviewers who often agree with others’ decisions

Anti-patterns in Modern Code Review: Symptoms and Prevalence

“…Recently, Hirao et al [16] studied patches with divergent review scores and found that 15%-37% of patches that receive multiple review scores suffer from divergent scores. Furthermore, [8] showed that a low level of agreement is more likely to take a longer reviewing time and discussion length. Thongtanunam et al [21] investigated patches that do not attract reviewers, not discussed, and receive slow initial feedback.…”

“…2) Divergent reviewers (DR): A code patch under review can suffer from divergent reviewers when reviewers cannot agree on the final evaluation by providing conflicting reviews and scores [16]. DR can lead to several problems in the review process including developer abandonment [11], poor team performance [29] and slow integration processes [8].…”

Modern code review

“…In the context of tool-based reviews, researchers have investigated the relationship between code change acceptance or response time and features of the changed code and authors [41], as well as the agreement among reviewers [22]. Qualitative investigations have been also conducted to define what constitutes a good code review according to industrial [11] and OSS developers [26].…”

How are IF-Conditional Statements Fixed Through Peer CodeReview?

“…Rigby et al [17] found that six large-scale OSS projects needed approximately one month to integrate a patch. Reviewers may disagree with one another and take even longer for discussion [18]. The process also requires identifying appropriate reviewers for each patch.…”