Watch Out for Extrinsic Bugs! A Case Study of Their Impact in Just-In-Time Bug Prediction Models on the OpenStack Project

Abstract

Intrinsic bugs are bugs for which a bug-introducing change can be identified in the version control system of a software. In contrast, extrinsic bugs are caused by external changes to a software, such as errors in external APIs; thereby they do not have an explicit bug-introducing change in the version control system. Although most previous research literature has assumed that all bugs are of intrinsic nature, in a previous study, we show that not all bugs are intrinsic. This paper shows an example of how considering extrinsic bugs can affect software engineering research. Specifically, we study the impact of extrinsic bugs in Just-In-Time bug prediction by partially replicating a recent study by McIntosh and Kamei on JIT models. These models are trained using properties of earlier bug-introducing changes. Since extrinsic bugs do not have bug-introducing changes in the version control system, we manually curate McIntosh and Kamei's dataset to distinguish between intrinsic and extrinsic bugs. Then, we address their original research questions, this time removing extrinsic bugs, to study whether bug-introducing changes are a moving target in Just-In-Time bug prediction. Finally, we study whether characteristics of intrinsic and extrinsic bugs are different. Our results show that intrinsic and extrinsic bugs are of different nature. When removing extrinsic bugs the performance is different up to 16 % Area Under the Curve points. This indicates that our JIT models obtain a more accurate representation of the real world. We conclude that extrinsic bugs negatively impact Just-In-Time models. Furthermore, we offer evidence that extrinsic bugs should be further investigated, as they can significantly impact how software engineers understand bugs.

Enhancing the defectiveness prediction of methods and classes via JIT

“…While countless studies investigated how to predict the defectiveness of commits (Herbold et al 2020;McIntosh and Kamei 2018;Pascarella et al 2019;Herbold 2019;Kondo et al 2020;Huang et al 2019;Fan et al 2019;Tu et al 2020;Rodriguezperez et al 2020;Pascarella et al 2020;Giger et al 2012), or classes Tantithamthavorn et al 2016bTantithamthavorn et al , 2019Tantithamthavorn et al , 2020Bennin et al 2018Bennin et al , 2019Herbold et al 2017Herbold et al , 20182019;Hosseini et al 2019;Yan et al 2017;Liu et al 2017;Chi et al 2017;Jing et al 2017;Di Nucci et al 6 5 2018; Palomba et al 2019;Song et al 2019;Zhang et al 2016Zhang et al , 2017Lee et al 2016;Yu et al 2019a;Peters et al 2019;Qu et al 2021;Shepperd et al 2018;Amasaki 2020;Bangash et al 2020;Kondo et al 2019;Morasca and Lavazza 2020;Mori and Uchihira 2019;Tian et al 2015;Jiarpakdee et al 2020;…”

“…During the testing phase, developers work to identify and eventually fix defects in the code before these defects can reach the deployment phase and, hence, become production defects. The focus of our paper is not on the Just in Time (JIT) defect predictions, which are usually performed during the development phase (Herbold 2019(Herbold , 2020McIntosh and Kamei 2018;Pascarella et al 2019;Kondo et al 2020;Huang et al 2019;Fan et al 2019;Tu et al 2020;Rodriguezperez et al 2020).…”

“…In the defect prediction area, researchers have proposed the use of different models to predict defects, leveraging, for example, product metrics (Basili et al 1996;Gyimóthy et al 2005;Khoshgoftaar et al 1996;Nagappan and Ball 2005;Hassan 2009), process metrics (Moser et al 2008), knowledge from where previous defects occurred (Ostrand et al 2005;Kim et al 2007), information about change-inducing fixes (Kim et al 2008;Fukushima et al 2014) and, recently, deep learning techniques to automatically engineer features from source code elements (Wang et al 2016). While many studies investigated how to predict the defectiveness of commits (Herbold et al 2019(Herbold et al , 2020McIntosh and Kamei 2018;Pascarella et al 2019Pascarella et al , 2020Kondo et al 2020;Huang et al 2019;Fan et al 2019;Tu et al 2020;Rodriguezperez et al 2020;Giger et al 2012), or classes Tantithamthavorn et al 2016bTantithamthavorn et al , 2019Tantithamthavorn et al , 2020Bennin et al 2018Bennin et al , 2019Herbold et al 2017Herbold et al , 2018Herbold et al , 2019Hosseini et al 2019;Yan et al 2017;Liu et al 2017;Chi et al 2017;Jing et al 2017;Di Nucci et al 2018;Palomba et al 2019;Song et al 2019;…”

On effort-aware metrics for defect prediction

“…Past studies investigated how to use defect prediction models, aka classifiers, to predict the defectiveness of different types of entities including commits (Fan et al 2021;Giger et al 2012;Rodríguez-Pérez et al 2020;Tu et al 2020), classes Bangash et al 2020;Chen et al 2020;Chi et al 2017;Herbold et al , 2019Hosseini et al 2019;Jiarpakdee et al 2020;Lee et al 2016;Liu et al 2017;Nucci et al 2018;Qu et al 2021a;Shepperd et al 2018;Tantithamthavorn et al 2016cYan et al 2017; or methods ) by leveraging, for example, product metrics (Basili et al 1996;Gyimóthy et al 2005;Khoshgoftaar et al 1996;Nagappan and Ball 2005;Hassan 2009), process metrics (Moser et al 2008), knowledge from where previous defects occurred (Ostrand et al 2005;Kim et al 2007), information about change-inducing fixes (Kim et al 2008;Fukushima et al 2014) and, recently, deep learning techniques to automatically engineer features from source code elements .…”

ApacheJIT: A Large Dataset for Just-In-Time Defect Prediction

“…Over the past few years, supervised deep learning models found their way to JIT defect prediction [9,10]. Although deep learning models have demonstrated solid performances in other areas of computing [1,4,16], DeepJIT [10] and and CC2Vec [9] do not outperform simple linear regression models [13,17]. This can be attributed to two main reasons.…”