Empirical Study: How Issue Classification Influences Software Defect Prediction

Abstract

Software defect prediction aims to identify potentially defective software modules to better allocate limited quality assurance resources. Practitioners often do this by utilizing supervised models trained using historical data. This data is gathered by mining version control and issue tracking systems. Version control commits are linked to issues they address. If the linked issue is classified as a bug report, the change is considered as bug fixing. The problem arises from the fact that issues are often incorrectly classified within issue tracking systems. This introduces noise into the gathered datasets. In this paper, we investigate the influence issue classification has on software defect prediction dataset quality and resulting model performance. To do this, we mine data from 7 popular open-source repositories, create issue classification and software defect prediction datasets for each of them. We investigate issue classification using four different methods; a simple keyword heuristic, an improved keyword heuristic, the FastText model and the RoBERTa model. Our results show that using the RoBERTa model for issue classification produces the best software defect prediction datasets, containing on average 14.3641% of mislabeled instances. SDP models trained on such datasets achieve superior performance, to those trained on SDP datasets created using other issue classification methods, in 65 out of 84 experiments, with 55 of them being statistically relevant. Furthermore, in 17 out of 28 experiments we could not show a statistically relevant performance difference between SDP models trained on RoBERTa derived software defect prediction datasets and those created using manually labeled issues.

Rand-Index Target Projective Gradient Deep Belief Network for Software Fault Prediction

“…A case study of software defect prediction to explore feasibility of employing static software metrics were investigated in [11]. Yet another empirical study on the mechanism for software defect prediction using classification was discussed in [12].…”

FSDP: Frequent Software Defects Prediction Based on Defect Correlation Learning for Quality Software Development

“…The creations of SDs are due to the coding mistakes, incorrect conditions, or flawed design. The impact of these defects is substantial, affecting the reliability and quality of the software [1]. Reliable software is expected to perform consistently and correctly under various conditions.…”

“…Identifying defects in the early stages helps in preventing downstream issues that might require extensive rework and resources. So, addressing issues at the inception of development prevents the accumulation of defects, resulting in a more robust and reliable system [1].…”

Graph-Driven Exploration of Issue Handling Schemes in Software Projects

“…Software defect predictions can be performed within a project or across project scopes. Studies at the level of project versions are also possible [32], where a prediction model uses historical data derived by mining version control and issue tracking systems. The impact of feature selection and sampling techniques on the accuracy of software fault prediction model is discussed in [33].…”

“…In the opposite case, a new vertex is added (lines 35-39). Next, we check for the existence of an edge between the subsequent path states S(i) and S(i + 1) in the created graph G. In the positive case, it updates the issue flow for this edge (E.count), and in the opposite case, a new edge is created (lines [24][25][26][27][28][29][30][31][32]. Having traced all states in the analyzed path of the considered issue, we check whether it is compatible (the same structure) with any previously analyzed paths.…”