Don't Disturb Me: Challenges of Interacting with Software Bots on Open Source Software Projects

Abstract

Software bots are used to streamline tasks in Open Source Software (OSS) projects' pull requests, saving development cost, time, and effort. However, their presence can be disruptive to the community. We identified several challenges caused by bots in pull request interactions by interviewing 21 practitioners, including project maintainers, contributors, and bot developers. In particular, our findings indicate noise as a recurrent and central problem. Noise affects both human communication and development workflow by overwhelming and distracting developers. Our main contribution is a theory of how human developers perceive annoying bot behaviors as noise on social coding platforms. This contribution may help practitioners understand the effects of adopting a bot, and researchers and tool designers may leverage our results to better support human-bot interaction on social coding platforms.

Dependency management bots in open-source systems—prevalence and adoption

“…In turn, most of the discussion around discarding those types of bots revolved around the noise that the bot generates. Some of those factors, such as noise ( Wessel et al, 2021 ) or the benefits in handling tasks at scale ( Erlenhov, de Oliveira Neto & Leitner, 2020 ), have also been seen in other studies as relevant factors to, respectively, hinder or improve the development workflow.…”

“…We mitigate disagreement between coders by (i) using few and fixed labels for the PRs conversations and (ii) using definitions from literature to label the content of discussions. Examples of (ii) are the list of themes related to the benefits of using bots from Erlenhov, de Oliveira Neto & Leitner (2020) or the definition of noise created by bots as proposed by Wessel et al (2021) . Moreover, creating distinct categories of code labels to capture the context of the PR conversation vs the content of the discussion allowed us to relate the discussions to the factors listed in RQ3.…”

“…They found that social exposure, competition, and observability affect the adoption. In a recent paper by Wessel et al (2021) , the initial interview study revealed several adoption challenges such as discoverability issues and configuration issues. The study then continues to discuss noise and introduces a theory about how certain behaviours of a bot can be perceived as noise.…”

Quality gatekeepers: investigating the effects of code review bots on pull request activities

“…However, developers might also perceive bot comments as noise, which disrupts the conversation in the pull request. Thus, project members should be aware of these possible side effects since noise is a recurrent problem when adopting bots on pull requests (Wessel et al 2021). For instance, they might consider re-configuring the bot to avoid some behaviors, such as high frequency of actions-bots performing repetitive actions, such as creating numerous pull requests and leaving dozen of comments in a row-and comments verbosity-bots providing comments with dense information.…”

Together or Apart? Investigating a mediator bot to aggregate bot's comments on pull requests

“…On social coding platforms such as GitHub, developers are often overwhelmed by bot pull request notifications, which interrupt their workflow [1]. As pointed out by Wessel et al [1], as bots have become new voices in developers' conversations, they may overburden developers who already suffer from information overload when communicating online [2]. This problem is especially relevant for newcomers, who require special support during the onboarding process due to the barriers they face [3].…”