Preliminary comparison of techniques for dealing with imbalance in software defect prediction

Rodríguez, Daniel; Herraiz, Israel; Harrison, Rachel; Dolado, José Javier; Riquelme, José C.

doi:10.1145/2601248.2601294

Cited by 118 publications

(78 citation statements)

References 46 publications

(39 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Recent defect prediction studies have compared the impact of class rebalancing techniques on the perfor-mance of defect prediction models [34,45,63,65,68,78,90]. For example, Kamei et al [34] show the performance improvement of class rebalancing techniques on 2 defect datasets of proprietary systems.…”

Section: Introductionmentioning

confidence: 99%

The Impact of Class Rebalancing Techniques on the Performance and Interpretation of Defect Prediction Models

Tantithamthavorn

Hassan

Matsumoto

2020

IIEEE Trans. Software Eng.

220

142

View full text Add to dashboard Cite

Defect prediction models that are trained on class imbalanced datasets (i.e., the proportion of defective and clean modules is not equally represented) are highly susceptible to produce inaccurate prediction models. Prior research compares the impact of class rebalancing techniques on the performance of defect prediction models. Prior research efforts arrive at contradictory conclusions due to the use of different choice of datasets, classification techniques, and performance measures. Such contradictory conclusions make it hard to derive practical guidelines for whether class rebalancing techniques should be applied in the context of defect prediction models. In this paper, we investigate the impact of 4 popularly-used class rebalancing techniques on 10 commonly-used performance measures and the interpretation of defect prediction models. We also construct statistical models to better understand in which experimental design settings that class rebalancing techniques are beneficial for defect prediction models. Through a case study of 101 datasets that span across proprietary and open-source systems, we recommend that class rebalancing techniques are necessary when quality assurance teams wish to increase the completeness of identifying software defects (i.e., Recall). However, class rebalancing techniques should be avoided when interpreting defect prediction models. We also find that class rebalancing techniques do not impact the AUC measure. Hence, AUC should be used as a standard measure when comparing defect prediction models.

show abstract

Section: Introductionmentioning

confidence: 99%

The Impact of Class Rebalancing Techniques on the Performance and Interpretation of Defect Prediction Models

Tantithamthavorn

Hassan

Matsumoto

2020

IIEEE Trans. Software Eng.

220

142

View full text Add to dashboard Cite

show abstract

“…The different approaches to deal with imbalanced can be classified into data as sampling, cost-sensitive, ensemble approaches or hybrid approaches [11]. In this work, we compare only rule or tree classifiers taking into account the imbalance nature of software defect datasets.…”

Section: Previous Workmentioning

confidence: 99%

The Consolidated Tree Construction algorithm in imbalanced defect prediction datasets

Ibarguren

Pérez

Mugerza

et al. 2017

2017 IEEE Congress on Evolutionary Computation (CEC)

Self Cite

View full text Add to dashboard Cite

In this short paper, we compare well-known rule/tree classifiers in software defect prediction with the CTC decision tree classifier designed to deal with class imbalance. It is well-known that most software defect prediction datasets are highly imbalance (non-defective instances outnumber defective ones). In this work, we focused only on tree/rule classifiers as these are capable of explaining the decision, i.e., describing the metrics and thresholds that make a module error prone. Furthermore, rules/decision trees provide the advantage that they are easily understood and applied by project managers and quality assurance personnel. The CTC algorithm was designed to cope with class imbalance and noise datasets instead of using preprocessing techniques (oversampling or undersampling), ensembles or cost weights of misclassification. The experimental work was carried out using the NASA datasets and results showed that induced CTC decision trees performed better or similar to the rest of the rule/tree classifiers.

show abstract

“…2014 IEEE Xplore Table 3. Continue Felderer et al (2014) On the role of defect taxonomy types for testin 2014 IEEE Xplore requirements: Results of a controlled experiment Rodriguez et al (2014) Preliminary comparison of techniques for dealing with 2014 ACM Digital Library imbalance in software defect prediction Femmer et al (2014) Rapid requirements checks with requirements smells: 2014 ACM Digital Library Two case studies Yusop et al (2016) Reporting usability defects: Do reporters report what 2016 ACM Digital Library software developers need? Cavezza et al (2014) Reproducibility of environment-dependent software 2014 IEEE Xplore failures: An experience report Langenfeld et al (2016) Requirements defects over a project lifetime: An 2016 Compendex SpringerLink empirical analysis of defect data from a 5-year automotive project at bosch Saito et al (2014) RISDM: A requirements inspection systems design 2014 IEEE Xplore methodology -perspective-based design of the pragmatic quality model and question set to SRS Travassos (2014) Software defects: Stay away from them.…”

Section: Data Sources X Rqs X Research Typesmentioning

confidence: 99%

Defect Types and Software Inspection Techniques: A Systematic Mapping Study

OliveiraJr¹,

Geraldi²

2017

Journal of Computer Science

View full text Add to dashboard Cite

Software inspection has been used to guarantee and control the quality of products by detecting defects, which can be spread out throughout the entire software life cycle. Therefore, the main premise is to identify and reduce the number of defect types in software artifacts during inspections. This work focuses on providing an up to dated overview of existing defects in the context of software inspection techniques. A systematic mapping was carried out, from which 2096 primary studies were retrieved and 32 were final selected. From the analysis, classification and aggregation of the retrieved studies, important different defect types were identified. Most studies encompass defect types by means of experiments and proposed techniques and approaches. Thus, as a main result, the identification of several different studies with distinct proposals concerned on defect types is evident. Although researchers have conducted studies over time, a general pattern on the detection of defects could not be established. Therefore, the scenario in which this study was carried out provides researchers with the capability of conducting further research in a motivating and challenging research topic, as well as practitioners with the adoption of empirically evaluated inspection techniques and respective defect types.

show abstract

Preliminary comparison of techniques for dealing with imbalance in software defect prediction

Cited by 118 publications

References 46 publications

The Impact of Class Rebalancing Techniques on the Performance and Interpretation of Defect Prediction Models

The Impact of Class Rebalancing Techniques on the Performance and Interpretation of Defect Prediction Models

The Consolidated Tree Construction algorithm in imbalanced defect prediction datasets

Defect Types and Software Inspection Techniques: A Systematic Mapping Study

Contact Info

Product

Resources

About