How bugs are born: a model to identify how bugs are introduced in software components

Abstract

When identifying the origin of software bugs, many studies assume that "a bug was introduced by the lines of code that were modified to fix it". However, this assumption does not always hold and at least in some cases, these modified lines are not responsible for introducing the bug. For example, when the bug was caused by a change in an external API. The lack of empirical evidence makes it impossible to assess how important these cases are and therefore, to which extent the assumption is valid. To advance in this direction, and better understand how bugs "are born", we propose a model for defining criteria to identify the first snapshot of an evolving software system that exhibits a bug. This model, based on the perfect test idea, decides whether a bug is observed after a change to the software. Furthermore, we studied the model's criteria by carefully analyzing how 116 bugs were introduced in two different open source software projects. The manual analysis helped classify the root cause of those bugs and created manually curated datasets with bug-introducing changes and with bugs that were not introduced by any change in the source code. Finally, we used these datasets to evaluate the performance of four existing SZZ-based algorithms for detecting bug-introducing changes. We found that SZZ-based algorithms are not very accurate, especially when multiple commits are found; the F-Score varies from 0.44 to 0.77, while the percentage of true positives does not exceed 63%. Our results show empirical evidence that the prevalent assumption, "a bug was introduced by the lines of code that were modified to fix it", is just one case of how bugs are introduced in a software system. Finding what introduced a bug is not trivial: bugs can be introduced by the developers and be in the code, or be created irrespective of the code. Thus, further research towards a better understanding of the origin of bugs in software projects could help to improve design integration tests and to design other procedures to make software development more robust. Keywords Bug origins • Bug-introducing changes • First-failing change • SZZ algorithm • Extrinsic bugs • Intrinsic bugs Communicated by: Per Runeson

Watch Out for Extrinsic Bugs! A Case Study of Their Impact in Just-In-Time Bug Prediction Models on the OpenStack Project

“…The percentage of mislabeled bugs is similar to the percentage reported in previous studies, ranging from 33% to 40% [30], [33]. Furthermore, the percentage of extrinsic bugs is also similar to the one reported in previous studies (9-21%) [2].…”

“…Education We believe that there is currently a strong bias towards training future software engineers exclusively on intrinsic bugs when identifying the origin of bugs as previous studies do not consider the extrinsic nature of bugs [1], [2]. Our findings suggest that we should educate students in the fact that software bugs do not always have their origin in a change in the VCS.…”

“…For example, a new search could be how to use natural language processing techniques in combination with deep learning techniques to classify bugs as extrinsic or intrinsic based on the textual information from the bug reports. Also, another research line could study different techniques to automate as much as possible the theoretical model proposed by Rodríguez-Pérez et al [2] to identify extrinsic and intrinsic bugs. We envision that these tools might be of benefit in other fields of software engineering such as testing/verification, software analytics and software maintenance and evolution.…”

“…Thus, we removed the changeIDs that were not BICs. Since extrinsic bugs do not have BICs, we removed the link between issueIDs classified as extrinsic bugs and their BICs following the recommendation of Rodríguez-Pérez et al [2]. Besides, we ignored changes that modified either at least 10,000 lines ("too much churn") or 100 files ("too many files") as they were likely no BICs.…”

“…R Ecent studies show that bugs do not have the same origin [1], [2]. While some bugs have their origin in explicit changes recorded in the version control system (VCS) of the software, other bugs are due to external changes that are not recorded in the VCS, e.g., changes in external APIs, compatibility changes or even changes in the specifications.…”