Empirical evaluation of defect projection models for widely-deployed production software systems

Abstract

Defect-occurrence projection is necessary for the development of methods to mitigate the risks of software defect occurrences. In this paper, we examine user-reported software defectoccurrence patterns across twenty-two releases of four widelydeployed, business-critical, production, software systems: a commercial operating system, a commercial middleware system, an open source operating system (OpenBSD), and an open source middleware system (Tomcat). We evaluate the suitability of common defect-occurrence models by first assessing the match between characteristics of widely-deployed production software systems and model structures. We then evaluate how well the models fit real world data. We find that the Weibull model is flexible enough to capture defect-occurrence behavior across a wide range of systems. It provides the best model fit in 16 out of the 22 releases. We then evaluate the ability of the moving averages and the exponential smoothing methods to extrapolate Weibull model parameters using fitted model parameters from historical releases. Our results show that in 50% of our forecasting experiments, these two naïve parameterextrapolation methods produce projections that are worse than the projection from using the same model parameters as the most recent release. These findings establish the need for further research on parameter-extrapolation methods that take into account variations in characteristics of widely-deployed, production, software systems across multiple releases.

Extracting software static defect models using data mining

“…Dynamic defects models follow distribution patterns including Rayleigh distribution and other similar curves [40,41,42]. Ref.…”

A method for forecasting defect backlog in large streamline software development projects and its industrial evaluation

“…Li et al [26] have shown that in the context of post-release defects the naïve models like moving average are not sufficient, but Weibull and Rayleigh models render more accurate results. In the light of this we recommend to use the results of Li et al when predicting post-release defect inflow and our results/forecasting model when working with pre-release defects.…”

Reliability analysis based on jump diffusion models for an open source cloud computing

“…On the other hand, the effective methods of dynamic testing management for new distributed development paradigm as typified by the open source project have been only a few presented [3], [4], [5], [6]. Also, there are some interesting research papers in terms of the cloud hardware, cloud service, and cloud performance evaluation [7], [8].…”