Controlling overfitting in software quality models: experiments with regression trees and classification

Khoshgoftaar, Taghi M.; Allen, Edward B.; Deng, Jianyu

doi:10.1109/metric.2001.915528

Cited by 18 publications

(18 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Koshgoftaar et al suggest balancing the Type 1 and Type 2 errors by empirical assessment of the most appropriate cut-off level for allocating projects to the success and failure categories [29]. However, an alternative procedure is to use a cut-off probability corresponding to the actual proportion of successful projects.…”

Section: Testmentioning

confidence: 99%

Evaluating logistic regression models to estimate software project outcomes

Cerpa

Bardeen

Kitchenham

et al. 2010

Information and Software Technology

View full text Add to dashboard Cite

Section: Testmentioning

confidence: 99%

Evaluating logistic regression models to estimate software project outcomes

Cerpa

Bardeen

Kitchenham

et al. 2010

Information and Software Technology

View full text Add to dashboard Cite

“…For example, Khoshgoftaar et al [9] classify modules as risky (will contain at least one field defect) or not risky (no field defects) for changed modules.…”

Section: Type Of Modelingmentioning

confidence: 99%

“…Product metrics have been shown to be important predictors by studies such as Khoshgoftaar et al [9].…”

Section: Categories Of Predictorsmentioning

confidence: 99%

See 1 more Smart Citation

Experiences and results from initiating field defect prediction and product test prioritization efforts at ABB Inc.

Herbsleb

Shaw

et al. 2006

Proceedings of the 28th International Conference on Software Engineering

View full text Add to dashboard Cite

Quantitatively-based risk management can reduce the risks associated with field defects for both software producers and software consumers. In this paper, we report experiences and results from initiating risk-management activities at a large systems development organization. The initiated activities aim to improve product testing (system/integration testing), to improve maintenance resource allocation, and to plan for future process improvements. The experiences we report address practical issues not commonly addressed in research studies: how to select an appropriate modeling method for product testing prioritization and process improvement planning, how to evaluate accuracy of predictions across multiple releases in time, and how to conduct analysis with incomplete information. In addition, we report initial empirical results for two systems with 13 and 15 releases. We present prioritization of configurations to guide product testing, field defect predictions within the first year of deployment to aid maintenance resource allocation, and important predictors across both systems to guide process improvement planning. Our results and experiences are steps towards quantitatively-based risk management.

show abstract

“…The application of ECM for software quality classification modeling was initially investigated by Khoshgoftaar et. al. in the context of controlling the overfitting tendencies of classification trees (Khoshgoftaar, Allen and Deng, 2001). By incorporating the costs of misclassifications in addition to the error rates, ECM (Johnson and Wichern, 1992) provides a more practical insight into model-performance as compared to the error rates by themselves.…”

Section: Modified Expected Cost Of Misclassificationmentioning

confidence: 99%

Resource oriented selection of rule-based classification models: An empirical case study

2006

View full text Add to dashboard Cite

The amount of resources allocated for software quality improvements is often not enough to achieve the desired software quality. Software quality classification models that yield a risk-based quality estimation of program modules, such as fault-prone (fp) and not fault-prone (nfp), are useful as software quality assurance techniques. Their usefulness is largely dependent on whether enough resources are available for inspecting the fp modules. Since a given development project has its own budget and time limitations, a resource-based software quality improvement seems more appropriate for achieving its quality goals. A classification model should provide quality improvement guidance so as to maximize resource-utilization.We present a procedure for building software quality classification models from the limited resources perspective. The essence of the procedure is the use of our recently proposed Modified Expected Cost of Misclassification (MECM) measure for developing resource-oriented software quality classification models. The measure penalizes a model, in terms of costs of misclassifications, if the model predicts more number of fp modules than the number that can be inspected with the allotted resources. Our analysis is presented in the context of our Rule-Based Classification Modeling (RBCM) technique. An empirical case study of a large-scale software system demonstrates the promising results of using the MECM measure to select an appropriate resource-based rule-based classification model.

show abstract

Controlling overfitting in software quality models: experiments with regression trees and classification

Cited by 18 publications

References 34 publications

Evaluating logistic regression models to estimate software project outcomes

Evaluating logistic regression models to estimate software project outcomes

Experiences and results from initiating field defect prediction and product test prioritization efforts at ABB Inc.

Resource oriented selection of rule-based classification models: An empirical case study

Contact Info

Product

Resources

About