Detecting and Characterizing Bots that Commit Code

Abstract

Background: Some developer activity traditionally performed manually, such as making code commits, opening, managing, or closing issues is increasingly subject to automation in many OSS projects. Specifically, such activity is often performed by tools that react to events or run at specific times. We refer to such automation tools as bots and, in many software mining scenarios related to developer productivity or code quality, it is desirable to identify bots in order to separate their actions from actions of individuals. Aim: Find an automated way of identifying bots and code committed by these bots, and to characterize the types of bots based on their activity patterns. Method and Result: We propose BIMAN, a systematic approach to detect bots using author names, commit messages, files modified by the commit, and projects associated with the commits. For our test data, the value for AUC-ROC was 0.9. We also characterized these bots based on the time patterns of their code commits and the types of files modified, and found that they primarily work with documentation files and web pages, and these files are most prevalent in HTML and JavaScript ecosystems. We have compiled a shareable dataset containing detailed information about 461 bots we found (all of which have more than 1000 commits) and 13,762,430 commits they created.

How Gamification Affects Software Developers: Cautionary Evidence from a Natural Experiment on GitHub

“…Also bots make a significant number of contributions, so it is important to filter them out carefully. Our end sample excludes over 99% of the bots identified in a recent paper on GitHub bot detection, which we discovered after our analyses were completed [47]. We kept the remaining developers who had at least 100 commits, were assigned a "USR" type (excluding organization accounts), and who had an associated geolocation from GHTorrent, leaving 433,138 developers.…”

Representation of Developer Expertise in Open Source Software

“…Further application of our approach might include: a) detecting if a developer is actually a bot by analyzing the concentration of their skill vector (similar to [49], [50]); b) checking the alignment between skill vectors of different developers for identity resolution (similar to [47]); c) analyzing the skill vectors of the developers in a project to infer the transparency of the corresponding software supply chain [51], [52], [53], [54], [55], [56].…”

Does the First-Response Matter for Future Contributions? A Study of First Contributions

“…(II) Bots. Bots [21] may threaten our calculation of text analysis, e.g., sentiment, toxicity, and emotion. We mitigate this by adopting a tool to identify bots in the OSS projects [11].…”