Cross-version defect prediction: use historical data, cross-project data, or both?

Amasaki, Sousuke

doi:10.1007/s10664-019-09777-8

Cited by 41 publications

(37 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the Cross-version context, we use the same Random Forest learner as the Within-project context. We do so as Amasaki [2] recently found that the difference in performance between the model constructed with the Random Forest learner and the top performing model in the Cross-version context (LACE2+NaiveBayes proposed by Peters et al [67]) to be small. Moreover, the Random Forest model is more accessible to practitioner and simpler than the LACE2+NaiveBayes model.…”

Section: Step 4: Model Constructionmentioning

confidence: 99%

Revisiting the Impact of Dependency Network Metrics on Software Defect Prediction

Gong,

Rajbahadur,

Hassan

et al. 2022

Preprint

View full text Add to dashboard Cite

Software dependency network metrics extracted from the dependency graph of the software modules by the application of Social Network Analysis (SNA metrics) have been shown to improve the performance of the Software Defect prediction (SDP) models. However, the relative effectiveness of these SNA metrics over code metrics in improving the performance of the SDP models has been widely debated with no clear consensus. Furthermore, some of the common SDP scenarios like predicting the number of defects in a module (Defect-count) in Cross-version and Cross-project SDP contexts remain unexplored. Such lack of clear directive on the effectiveness of SNA metrics when compared to the widely used code metrics prevents us from potentially building better performing SDP models. Therefore, through a case study of 9 open source software projects across 30 versions, we study the relative effectiveness of SNA metrics when compared to code metrics across 3 commonly used SDP contexts (Within-project, Cross-version and Crossproject) and scenarios (Defect-count, Defect-classification (classifying if a module is defective) and Effort-aware (ranking the defective modules w.r.t to the involved effort)). We find the SNA metrics by themselves or along with code metrics improve the performance of SDP models over just using code metrics on 5 out of the 9 studied SDP scenarios (three SDP scenarios across three SDP contexts). However, we note that in some cases the improvements afforded by considering SNA metrics over or alongside code metrics might only be marginal, whereas in other cases the improvements could be potentially large. Based on these findings we suggest that the future work should: consider SNA metrics alongside code metrics in their SDP models; as well as consider Ego metrics and Global metrics, the two different types of the SNA metrics separately when training SDP models as they behave differently.

show abstract

Section: Step 4: Model Constructionmentioning

confidence: 99%

Revisiting the Impact of Dependency Network Metrics on Software Defect Prediction

Gong,

Rajbahadur,

Hassan

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We considered traditional two-class classification techniques widely used in previous studies (Hall et al, 2011) as our aim is to benchmark one-class predictors vs. traditional two-class predictors, and not to search for the best prediction technique. If OCSVM is not able to outperform such traditional baselines, it is reasonable to assume that it will not perform better than more sophisticated ones proposed for cross-version and cross-project defect prediction (e.g., (Nam et al, 2013;Xia et al, 2016;Herbold et al, 2018;Zhou et al, 2018;Hosseini et al, 2019;Amasaki, 2020)).…”

Section: Threats To Validitymentioning

confidence: 99%

Investigating the Use of One-Class Support Vector Machine for Software Defect Prediction

Moussa¹,

Azar²,

Sarro³

2022

Preprint

View full text Add to dashboard Cite

Early software defect identification is considered an important step towards software quality assurance. Software defect prediction aims at identifying software components that are likely to cause faults before a software is made available to the end-user. To date, this task has been modeled as a twoclass classification problem, however its nature also allows it to be formulated as a one-class classification task.Preliminary results obtained in prior work show that One-Class Support Vector Machine (OCSVM) can outperform two-class classifiers for defect prediction. If confirmed, these results would overcome the data imbalance problem researchers have for long attempted to tackle in this field.In this paper, we further investigate whether learning from one class only is sufficient to produce effective defect prediction models by conducting a thorough large-scale empirical study investigating 15 real-world software projects, three validation scenarios, eight classifiers, robust evaluation measures and statistical significance tests. The results reveal that OCSVM is more suitable for cross-version and cross-project, rather than for within-project defect prediction, thus suggesting it performs better with heterogeneous data. While, we cannot conclude that OCSVM is the best classifier (Random Forest performs best herein), our results show interesting findings that open up further research avenues for training accurate defect prediction classifiers when defective instances are scarce or unavailable.

show abstract

“…The Cross-version defect prediction scenario is one of the actively studied scenarios in within-project defect prediction [49]- [51]. In this paper, to perform a cross-version defect prediction scenario, for each project, we use its latest version as the test version and randomly select two earlier versions as the training data to build defect prediction models respectively.…”

Section: Cross-version Defect Predictionmentioning

confidence: 99%

Explainable Software Defect Prediction: Are We There Yet?

2021

View full text Add to dashboard Cite

Explaining the results of defect prediction models is practical but challenging to achieve. Recently, Jiarpakdee et al. [1] proposed to use two state-of-the-art model-agnostic techniques (i.e., LIME and BreakDown) to explain prediction results. Their study showed that model-agnostic techniques can achieve remarkable performance, and the generated explanations can assist developers to understand the prediction results. However, the fact that they only examined both LIME and BreakDown in a single defect prediction setting calls into question the consistency and reliability of model-agnostic techniques on defect prediction models under various settings.In this paper, we set out to investigate the reliability and stability of explanation generation approaches based on model-agnostic techniques, i.e., LIME and BreakDown, on defect prediction models under different settings, e.g., data sampling techniques, machine learning classifiers, and prediction scenarios used when building defect prediction models. Specifically, we use both LIME and BreakDown to generate explanations for the same instance under various defect prediction models with different settings and then check the consistency of the generated explanations for the instance. We reused the same defect data from Jiarpakdee et al. in our experiments. The results show that both LIME and BreakDown generate inconsistent explanations under different defect prediction settings for the same test instances. These imply that the model-agnostic techniques are unreliable for practical explanation generation. In addition, our manual analysis shows that none of the generated explanations can reflect the root causes of the predicted defects, which further weakens the usefulness of model-agnostic based explanation generation. Overall, with this study, we urge a revisit of existing model-agnostic based studies in software engineering and call for more research in explainable defect prediction towards achieving reliable and stable explanation generation.

show abstract

Cross-version defect prediction: use historical data, cross-project data, or both?

Cited by 41 publications

References 65 publications

Revisiting the Impact of Dependency Network Metrics on Software Defect Prediction

Revisiting the Impact of Dependency Network Metrics on Software Defect Prediction

Investigating the Use of One-Class Support Vector Machine for Software Defect Prediction

Explainable Software Defect Prediction: Are We There Yet?

Contact Info

Product

Resources

About