Quality gatekeepers: investigating the effects of code review bots on pull request activities

Abstract

Software bots have been facilitating several development activities in Open Source Software (OSS) projects, including code review. However, these bots may bring unexpected impacts to group dynamics, as frequently occurs with new technology adoption. Understanding and anticipating such effects is important for planning and management. To analyze these effects, we investigate how several activity indicators change after the adoption of a code review bot. We employed a regression discontinuity design on 1,194 software projects from GitHub. We also interviewed 12 practitioners, including open-source maintainers and contributors. Our results indicate that the adoption of code review bots increases the number of monthly merged pull requests, decreases monthly non-merged pull requests, and decreases communication among developers. From the developers’ perspective, these effects are explained by the transparency and confidence the bot comments introduce, in addition to the changes in the discussion focused on pull requests. Practitioners and maintainers may leverage our results to understand, or even predict, bot effects on their projects.

On Wasted Contributions: Understanding the Dynamics of Contributor-Abandoned Pull Requests–A Mixed-Methods Study of 10 Large Open-Source Projects

“…Wessel et al [62] studied the impact of adopting code review bots and found that it accelerates the increasing trend in the number of merged PRs, accelerates the decreasing trend in the number of closed PRs, decreases the number of comments in all PRs, increases the resolution time of merged PRs, and reverses the increasing trend in the resolution time of closed PRs. In another study, Wessel et al [65] investigated the impact of adopting GitHub Actions and found that it decreases the number of both merged PRs and closed PRs, increases the number of discussion comments in merged PRs, decreases the number of discussion comments in closed PRs, increases the resolution time of merged PRs, and decreases the resolution time of closed PRs. However, our study focuses on investigating the impact of adopting Stale bot to better understand its helpfulness for pull-based development.…”

“…To investigate the impact of adopting Stale bot, we first need to quantify the performance of the pull-based development workflow in the studied projects. For this purpose, we consult similar studies that investigated the effect of an intervention on the pull-based development of opensource projects [62,65]. As shown in Table 3, we measure 13 performance indicators covering six dimensions: (1) resolved PRs, (2) review latency, (3) resolution time, (4) review discussion, (5) PR updates, and (6) contributor retention.…”

“…As shown in Table 3, we measure 13 performance indicators covering six dimensions: (1) resolved PRs, (2) review latency, (3) resolution time, (4) review discussion, (5) PR updates, and (6) contributor retention. Similar to previous studies [62,65], we differentiate between the indicators of merged PRs (ending with _m) and closed PRs (ending with _c) to take into account the inherent differences in the characteristics of accepted and rejected PRs. To identify merged PRs, we resort to heuristics [31] similar to [28] because accepted PRs are not always merged using the standard methods provided through the GitHub interface.…”

“…To estimate the potential impact of Stale bot on the indicators of the projects, we rely on interrupted time-series analysis (ITS) [58]. This method is a well-established quasi-experiment previously used in the software engineering literature to study the impact of an intervention (e.g., [62,65,73]). To estimate the longitudinal impact of an intervention (i.e., the adoption of Stale bot in our case), ITS compares a period before and after the intervention assuming the trend would be retained if the intervention had not occurred.…”

“…We measured 13 different performance indicators covering various dimensions, including resolved PRs, review latency, resolution time, review discussion, PR updates, and contributor retention. To identify these indicators, we consulted similar studies that investigated the effects of interventions on the pull-based development of projects [62,65] and also drew from our previous experience studying abandoned PRs [30]. Nevertheless, projects may experience changes in aspects that we did not consider or that are difficult to quantify, such as code quality.…”