The Power of Bots

Abstract

Leveraging the pull request model of social coding platforms, Open Source Software (OSS) integrators review developers' contributions, checking aspects like license, code quality, and testability. Some projects use bots to automate predefined, sometimes repetitive tasks, thereby assisting integrators' and contributors' work. Our research investigates the usage and impact of such bots. We sampled 351 popular projects from GitHub and found that 93 (26%) use bots. We classified the bots, collected metrics from before and after bot adoption, and surveyed 228 developers and integrators. Our results indicate that bots perform numerous tasks. Although integrators reported that bots are useful for maintenance tasks, we did not find a consistent, statistically significant difference between before and after bot adoption across the analyzed projects in terms of number of comments, commits, changed files, and time to close pull requests. Our survey respondents deem the current bots as not smart enough and provided insights into the bots' relevance for specific tasks, challenges, and potential new features. We discuss some of the raised suggestions and challenges in light of the literature in order to help GitHub bot designers reuse and test ideas and technologies already investigated in other contexts.

Scaling open source communities

“…In comparison to projects that do not use CI, these projects (i) release twice as often, (ii) accept pull requests faster, and (iii) developers are less worried about breaking the build [28]. Recent studies investigated bots seeking to automate repetitive tasks in the social collaboration platform such as GitHub [36,58,59]. For example, the Hound bot was invented to verify code style violations [58].…”

“…Recent studies investigated bots seeking to automate repetitive tasks in the social collaboration platform such as GitHub [36,58,59]. For example, the Hound bot was invented to verify code style violations [58].…”

Enhancing developers’ support on pull requests activities with software bots

“…Our results indicate that bots' adoption is indeed widespread in OSS projects hosted on GitHub. Bots perform several tasks, including ensuring license agreement signing, reporting continuous integration failures, reviewing code and pull requests, triaging issues, and refactoring source code [23]. We also openly asked contributors and maintainers about the "challenges of using bots" on pull requests.…”

Bot or not?

“…To this end, we identified bots based on a combination of different methods. First we gathered evidence of bots that were reported by other researchers [1,2,8,28,29] and websites. 1 We also checked for presence of the word "bot" in the name and profile information of each GitHub identity, as this is frequently the case [28].…”

“…In addition, projects are increasingly relying on automated bots that use one or more GitHub identities to carry out routine tasks and to interact with project members. Such bots have been shown to improve collaborative development [4,8,17,28], for example by improving software quality through automated refactorings [31], by generating patches for bugs [21], by supporting continuous integration [2] and by automatically closing abandoned issues and pull requests [29].…”