Finding Predictors of Field Defects for Open Source Software Systems in Commonly Available Data Sources: A Case Study of OpenBSD

Abstract

Open source software systems are important components of many business software applications. Field defect predictions for open source software systems may allow organizations to make informed decisions regarding open source software components. In this paper, we remotely measure and analyze predictors (metrics available before release) mined from established data sources (the code repository and the request tracking system) as well as a novel source of data (mailing list archives) for nine releases of OpenBSD. First, we attempt to predict field defects by extending a software reliability model fitted to development defects. We find this approach to be infeasible, which motivates examining metrics-based field defect prediction. Then, we evaluate 139 predictors using established statistical methods: Kendall's rank correlation, Pearson's rank correlation, and forward AIC model selection. The metrics we collect include product metrics, development metrics, deployment and usage metrics, and software and hardware configurations metrics. We find the number of messages to the technical discussion mailing list during the development period (a deployment and usage metric captured from mailing list archives) to be the best predictor of field defects. Our work identifies predictors of field defects in commonly available data sources for open source software systems and is a step towards metrics-based field defect prediction for quantitatively-based decision making regarding open source software components.

Modelling Failures Occurrences of Open Source Software with Reliability Growth

“…In our research, after selecting the appropriate open source projects and cleaning the data our work provides an extension of the work of Li et al ( [7]) and Succi et al…”

“…( [14]) combining the two approaches -software reliability growth across versions ( [7]) and reliability growth and measures of accuracy and prediction ( [14]) -and replicating the study on three different OSS. Mining repositories for software failures is a hot topic ( [1], [2], [5], [6], [7], [8]), but at our knowledge it does not seem to be any research that has compared results across multiple releases and multiple OSS through multiple measures of accuracy and prediction.…”

“…Mining repositories for software failures is a hot topic ( [1], [2], [5], [6], [7], [8]), but at our knowledge it does not seem to be any research that has compared results across multiple releases and multiple OSS through multiple measures of accuracy and prediction.…”

Empirical Analysis of the Complexity Evolution in Open-Source Software Systems

“…These studies can be classified into three different categories. The first category studied some different successful open-source projects, OpenBSD [16], FreeBSD [9], Apache [21], and Debian GNU/Linux [28]. The second category studied the similarities of the process used by successful open-source projects, Arla and Mozila Projects [4], Apache and Mozilla [22], and fifteen OSS Projects [32].…”

Statistical Analysis on Software Metrics Affecting Modularity in Open Source Software

“…First approach of the study is by studying several successful OSS Projects, i.e. the study on Debian GNU/Linux [28], FreeBSD [11], Apache [23], and OpenBSD [18]. The second approach is trying to find the similarity in the processes in many successful OSS Projects i.e.…”