The impact of human factors on the participation decision of reviewers in modern code review

Abstract

Modern Code Review (MCR) plays a key role in software quality practices. In MCR process, a new patch (i.e., a set of code changes) is encouraged to be examined by reviewers in order to identify weaknesses in source code prior to an integration into main software repositories. To mitigate the risk of having future defects, prior work suggests that MCR should be performed with sufficient review participation. Indeed, recent work shows that a low number of participated reviewers is associated with poor software quality. However, there is a likely case that a new patch still suffers from poor review participation even though reviewers were invited. Hence, in this paper, we set out to investigate the factors that are associated with the participation decision of an invited reviewer. Through a case study of 230,090 patches spread across the Android, LibreOffice, OpenStack and Qt systems, we find that (1) 16%-66% of patches have at least one invited reviewer who did not respond to the review invitation; (2) human factors play an important role in predicting whether or not an invited reviewer will participate in a review; (3) a review participation rate of an invited reviewers and code authoring experience of an invited reviewer are highly associated with the participation decision of an invited reviewer. These results can help practitioners better understand about how human factors associate with the participation decision of reviewers and serve as guidelines for inviting reviewers, leading to a better inviting decision and a better reviewer participation.

Investigating the Validity of Ground Truth in Code Reviewer Recommendation Studies

“…The discrepancy between real-life reviewer assignments and recommendation models raises a threat to the validity of the recommendation studies. The analysis of Ruangwan et al [36] with four large software projects also supports our idea of the problematic ground truth in reviewer data related to the availability reasons. In the future, we would like to introduce quantitative evidence for attribute substitution bias as well.…”

“…[11], [36], [37] Review experience [11], [12], [35] Code familiarity [36], [37] Patch characteristics [11], [35] Volunteer for review [11], [36 Greiler et al [35] complete a comprehensive study among Microsoft developers in order to find the critical aspects while deciding the reviewer. They find out that assigning a volunteer reviewer might be preferred rather than assigning an unwilling reviewer.…”

“…They find out that assigning a volunteer reviewer might be preferred rather than assigning an unwilling reviewer. Ruangwan et al [36] emphasizes the effect of the developer workload and patch characteristics on the reviewer selection process.…”

“…Code reviewer recommendation studies do not consider any kind of human aspects and assume that the ideal reviewer is the same as the assigned reviewer in real life. Nevertheless, in some studies [11], [12], [35], [36], it is clearly illustrated that the real-life decisions are not usually made with the goal of finding the ideal reviewer.…”

“…In order to provide further proof for problematic ground truth instances, an analysis of pull request reviews from four large-size software projects completed by Ruangwan et al [36] is provided. Through a case study of 230,090 patches spread across the Android, LibreOffice, OpenStack and Qt repositories, Ruangwan et al reveal that a remarkable number of patches (pull requests) have at least one non-responsive reviewer to the review request.…”