Supervised vs Unsupervised Models: A Holistic Look at Effort-Aware Just-in-Time Defect Prediction

Huang, Qiao; Xia, Xin; Lo, David

doi:10.1109/icsme.2017.51

Cited by 106 publications

(69 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They say a "good" defect predictor selects the 20% of files containing 80% of the defects In the literature, this 20/80 rule is often called P opt 20 (the percent of the bugs found after reading 20%). P opt 20 is widely used in the literature and, for details on that measure, we refer the reader to those publications [18], [42], [48], [62], [64], [69], [69], [111]. For this paper, all we need to say about P opt 20 is the conclusions reached from this metric are nearly the same as the conclusions reached via G-score.…”

Section: Evaluation Criteriamentioning

confidence: 98%

See 1 more Smart Citation

Better Data Labelling With EMBLEM (and how that Impacts Defect Prediction)

2022

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Standard automatic methods for recognizing problematic development commits can be greatly improved via the incremental application of human+artificial expertise. In this approach, called EMBLEM, an AI tool first explore the software development process to label commits that are most problematic. Humans then apply their expertise to check those labels (perhaps resulting in the AI updating the support vectors within their SVM learner). We recommend this human+AI partnership, for several reasons. When a new domain is encountered, EMBLEM can learn better ways to label which comments refer to real problems. Also, in studies with 9 open source software projects, labelling via EMBLEM's incremental application of human+AI is at least an order of magnitude cheaper than existing methods (≈ eight times). Further, EMBLEM is very effective. For the data sets explored here, EMBLEM better labelling methods significantly improved Popt20 and G-score performance in nearly all the projects studied here. TABLE 1This paper argues against using keywords like these as a method for labelling a commit as "buggy'.

show abstract

Section: Evaluation Criteriamentioning

confidence: 98%

“…This value is the harmonic mean between recall and falsealarm of risky software commit prediction power. There are other evaluation scores that could be applied to this kind of analysis [42] and, in the future, it would be useful to test in the central claim of this paper holds for more than just G-scores and P opt 20.…”

Section: Evaluation Biasmentioning

confidence: 99%

Better Data Labelling With EMBLEM (and how that Impacts Defect Prediction)

2022

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

show abstract

“…As this paper was going to press, we learned of new papers that updated the Yang [17]. We thank these authors for the courtesy of sharing a pre-print of those new results.…”

Section: Addendummentioning

confidence: 99%

Revisiting unsupervised learning for defect prediction

Wei

Menzies

2017

Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering

108

View full text Add to dashboard Cite

Collecting quality data from so ware projects can be time-consuming and expensive. Hence, some researchers explore "unsupervised" approaches to quality prediction that does not require labelled data. An alternate technique is to use "supervised" approaches that learn models from project data labelled with, say, "defective" or "notdefective". Most researchers use these supervised models since, it is argued, they can exploit more knowledge of the projects.At FSE'16, Yang et al. reported startling results where unsupervised defect predictors outperformed supervised predictors for e ort-aware just-in-time defect prediction. If con rmed, these results would lead to a dramatic simpli cation of a seemingly complex task (data mining) that is widely explored in the so ware engineering literature.is paper repeats and refutes those results as follows.(1) ere is much variability in the e cacy of the Yang et al. predictors so even with their approach, some supervised data is required to prune weaker predictors away. (2) eir ndings were grouped across N projects. When we repeat their analysis on a project-by-project basis, supervised predictors are seen to work be er.Even though this paper rejects the speci c conclusions of Yang et al., we still endorse their general goal. In our our experiments, supervised predictors did not perform outstandingly be er than unsupervised ones for e ort-aware just-in-time defect prediction. Hence, they may indeed be some combination of unsupervised learners to achieve comparable performance to supervised ones. We therefore encourage others to work in this promising area. KEYWORDSData analytics for so ware engineering, so ware repository mining, empirical studies, defect prediction ACM Reference format:

show abstract

“…Then, they attempted to combine complexity metrics with more metrics such as code churn metrics and token frequency metrics [26,31,43,47,48,52,54,54,57,58,58,65,79,81]. Then, advances have been made to use unsupervised machine learning to predict bugs [25,32,36,46,75,76,77,78,80] using the similar set of complexity metrics. These approaches use the similar metrics as those in bug prediction, but do not capture the difference between vulnerable code and buggy code, which hinders the effectiveness.…”

Section: Related Workmentioning

confidence: 99%

LEOPARD: Identifying Vulnerable Code for Vulnerability Assessment Through Program Metrics

Chen

et al. 2019

2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)

View full text Add to dashboard Cite

Identifying potentially vulnerable locations in a code base is critical as a pre-step for effective vulnerability assessment; i.e., it can greatly help security experts put their time and effort to where it is needed most. Metric-based and pattern-based methods have been presented for identifying vulnerable code. The former relies on machine learning and cannot work well due to the severe imbalance between non-vulnerable and vulnerable code or lack of features to characterize vulnerabilities. The latter needs the prior knowledge of known vulnerabilities and can only identify similar but not new types of vulnerabilities.In this paper, we propose and implement a generic, lightweight and extensible framework, LEOPARD, to identify potentially vulnerable functions through program metrics. LEOPARD requires no prior knowledge about known vulnerabilities. It has two steps by combining two sets of systematically derived metrics. First, it uses complexity metrics to group the functions in a target application into a set of bins. Then, it uses vulnerability metrics to rank the functions in each bin and identifies the top ones as potentially vulnerable. Our experimental results on 11 real-world projects have demonstrated that, LEOPARD can cover 74.0% of vulnerable functions by identifying 20% of functions as vulnerable and outperform machine learning-based and static analysis-based techniques. We further propose three applications of LEOPARD for manual code review and fuzzing, through which we discovered 22 new bugs in real applications like PHP, radare2 and FFmpeg, and eight of them are new vulnerabilities.

show abstract

Supervised vs Unsupervised Models: A Holistic Look at Effort-Aware Just-in-Time Defect Prediction

Cited by 106 publications

References 53 publications

Better Data Labelling With EMBLEM (and how that Impacts Defect Prediction)

Better Data Labelling With EMBLEM (and how that Impacts Defect Prediction)

Revisiting unsupervised learning for defect prediction

LEOPARD: Identifying Vulnerable Code for Vulnerability Assessment Through Program Metrics

Contact Info

Product

Resources

About