Application of a usage profile in software quality models

11th IEEE International Software Metrics Symposium (METRICS'05)

Open source software systems are important components of many business software applications. Field defect predictions for open source software systems may allow organizations to make informed decisions regarding open source software components. In this paper, we remotely measure and analyze predictors (metrics available before release) mined from established data sources (the code repository and the request tracking system) as well as a novel source of data (mailing list archives) for nine releases of OpenBSD. First, we attempt to predict field defects by extending a software reliability model fitted to development defects. We find this approach to be infeasible, which motivates examining metrics-based field defect prediction. Then, we evaluate 139 predictors using established statistical methods: Kendall's rank correlation, Pearson's rank correlation, and forward AIC model selection. The metrics we collect include product metrics, development metrics, deployment and usage metrics, and software and hardware configurations metrics. We find the number of messages to the technical discussion mailing list during the development period (a deployment and usage metric captured from mailing list archives) to be the best predictor of field defects. Our work identifies predictors of field defects in commonly available data sources for open source software systems and is a step towards metrics-based field defect prediction for quantitatively-based decision making regarding open source software components.

Section: Software and Hardware Configuration Metric (Computed Using Msupporting

confidence: 93%

“…Show that the predictor is selected using a model selection method. This method is used by Jones et al [6] and Mockus et al [22].…”

Section: Methods Of Establishing the Importance Of Predictorsmentioning

confidence: 99%

See 1 more Smart Citation

Finding Predictors of Field Defects for Open Source Software Systems in Commonly Available Data Sources: A Case Study of OpenBSD

11th IEEE International Software Metrics Symposium (METRICS'05)

“…Jones et. al at Nortel use the percentage of deployed systems with a module installed as a surrogate for usage along with software content measures to predict the likelihood that a module will be faulty in the field [10]. They correctly identify approximately 70% of the faulty modules while misidentifying less than 26% of the faulty modules.…”

Section: Related Workmentioning

confidence: 99%

Empirical evaluation of defect projection models for widely-deployed production software systems

SIGSOFT Softw. Eng. Notes

et al. 2004

Defect-occurrence projection is necessary for the development of methods to mitigate the risks of software defect occurrences. In this paper, we examine user-reported software defectoccurrence patterns across twenty-two releases of four widelydeployed, business-critical, production, software systems: a commercial operating system, a commercial middleware system, an open source operating system (OpenBSD), and an open source middleware system (Tomcat). We evaluate the suitability of common defect-occurrence models by first assessing the match between characteristics of widely-deployed production software systems and model structures. We then evaluate how well the models fit real world data. We find that the Weibull model is flexible enough to capture defect-occurrence behavior across a wide range of systems. It provides the best model fit in 16 out of the 22 releases. We then evaluate the ability of the moving averages and the exponential smoothing methods to extrapolate Weibull model parameters using fitted model parameters from historical releases. Our results show that in 50% of our forecasting experiments, these two naïve parameterextrapolation methods produce projections that are worse than the projection from using the same model parameters as the most recent release. These findings establish the need for further research on parameter-extrapolation methods that take into account variations in characteristics of widely-deployed, production, software systems across multiple releases.

“…Previous studies used either cross-validation [33] or random data with-holding [5]; however, neither procedure is realistic. In a realistic setting, only a non-random subset of data is actually available before release (i.e.…”

Section: Field Defect Prediction For Maintenance Resource Allocationmentioning

confidence: 99%

Experiences and results from initiating field defect prediction and product test prioritization efforts at ABB Inc.

Proceedings of the 28th International Conference on Software Engineering

et al. 2006

Quantitatively-based risk management can reduce the risks associated with field defects for both software producers and software consumers. In this paper, we report experiences and results from initiating risk-management activities at a large systems development organization. The initiated activities aim to improve product testing (system/integration testing), to improve maintenance resource allocation, and to plan for future process improvements. The experiences we report address practical issues not commonly addressed in research studies: how to select an appropriate modeling method for product testing prioritization and process improvement planning, how to evaluate accuracy of predictions across multiple releases in time, and how to conduct analysis with incomplete information. In addition, we report initial empirical results for two systems with 13 and 15 releases. We present prioritization of configurations to guide product testing, field defect predictions within the first year of deployment to aid maintenance resource allocation, and important predictors across both systems to guide process improvement planning. Our results and experiences are steps towards quantitatively-based risk management.