Better Data Labelling With EMBLEM (and how that Impacts Defect Prediction)

Abstract

Standard automatic methods for recognizing problematic development commits can be greatly improved via the incremental application of human+artificial expertise. In this approach, called EMBLEM, an AI tool first explore the software development process to label commits that are most problematic. Humans then apply their expertise to check those labels (perhaps resulting in the AI updating the support vectors within their SVM learner). We recommend this human+AI partnership, for several reasons. When a new domain is encountered, EMBLEM can learn better ways to label which comments refer to real problems. Also, in studies with 9 open source software projects, labelling via EMBLEM's incremental application of human+AI is at least an order of magnitude cheaper than existing methods (≈ eight times). Further, EMBLEM is very effective. For the data sets explored here, EMBLEM better labelling methods significantly improved Popt20 and G-score performance in nearly all the projects studied here. TABLE 1This paper argues against using keywords like these as a method for labelling a commit as "buggy'.

Evaluating SZZ Implementations Through a Developer-Informed Oracle

“…Davies et al [13] AG-SZZ [14] Manually defined (researchers) 3 174 MA-SZZ da Costa et al [14] AG-SZZ [6], [9], [10], [15], [16], [38] Automatically computed metrics 10 2,637 RA-SZZ Neto et al [15] MA-SZZ [5], [6], [15] Manually defined (researchers) For each one, we specify (i) the algorithm on which it is based, (ii) references of works using it, (iii) the oracle used in the evaluation (how it was built, number of projects and bug fixes considered).…”

“…For example, Aman et al [9] studied the role of local variable names in fault-introducing commits and they used SZZ to retrieve such commits, while Palomba et al [17] focused on the impact of code smells, and used SZZ to determine whether an artifact was smelly when a fault was introduced. Many studies also leverage SZZ to evaluate defect prediction approaches [2]- [6], [19], [21], [26], [34], [38].…”

DebtFree: Minimizing Labeling Cost in Self-Admitted Technical Debt Identification using Semi-Supervised Learning

“…In this work, we aim to better data generation associated with building models for SATDs identification by reducing the labeling effort come from manual method [67] and improving the labeling quality of fully automated methods [84]. Moreover, our investigation also showed that the effort-aware method we propose, DebtFree, also performs statistically similar or even better than two SOTA works [84,55].…”

“…Fig. 1: Workflows of DebtFree = Pseudo-Labeling (via Unsupervised Learning, i.e., CLA [47]) + Filtering (via CLA [47] or Jitterbug's Easy [84]) + Active Learning (via Emblem [67], Jitterbug's Hard [84], or this study's Falcon).…”

Defect Prediction Using Akaike and Bayesian Information Criterion

“…They found that the model trained on data labeled by B-SZZ and MA-SZZ does not cause a considerable reduction in accuracy but AG-SZZ causes a considerable decline in the accuracy. Tu et al [7] worked on better data labeling using a proposed approach known as EMBLEM. In this approach first AI labels the data and then human experts check the labels which in turn updates the AI.…”