Experience

Abstract

Data is a cornerstone of empirical software engineering (ESE) research and practice. Data underpin numerous process and project management activities, including the estimation of development effort and the prediction of the likely location and severity of defects in code. Serious questions have been raised, however, over the quality of the data used in ESE. Data quality problems caused by noise, outliers, and incompleteness have been noted as being especially prevalent. Other quality issues, although also potentially important, have received less attention. In this study, we assess the quality of 13 datasets that have been used extensively in research on software effort estimation. The quality issues considered in this article draw on a taxonomy that we published previously based on a systematic mapping of data quality issues in ESE. Our contributions are as follows: (1) an evaluation of the "fitness for purpose" of these commonly used datasets and (2) an assessment of the utility of the taxonomy in terms of dataset benchmarking. We also propose a template that could be used to both improve the ESE data collection/submission process and to evaluate other such datasets, contributing to enhanced awareness of data quality issues in the ESE community and, in time, the availability and use of higher-quality datasets.

Multi-Objective Software Effort Estimation: A Replication Study

“…This selection happens multiple time in each generation to fill the mating pool [65]. The tournament selector, on the other hand, is used to select the best n solutions (usually n ∈ [1,10]) to be copied straight into the next generation [89]. Each tournament randomly picks a number of solutions from the population and selects the fittest one.…”

“…We used the same five publicly available datasets used in the original study to empirically investigate our research questions. These five datasets, namely China, Desharnais, Finnish, Maxwell, and Miyazaki, have been extensively used in previous software effort estimation studies [10].…”

“…Table 1 summarises the descriptive statistics of the features of the datasets we considered. We observe that these datasets differ in terms of: observation number (from 38 to 499 projects); number and type of features (from 4 to 17 features, including a variety of features describing the software projects); companies involved, i.e., within-company (WC), or cross-company (CC); and geographical locations [10]. The software projects included in these datasets also differ in technical characteristics.…”

“…Since many of the samples comes from non-normally distributed populations [10], [89] a non-parametric method ought to be used as the statistical significance test. To this end, we use the Wilcoxon Rank-Sum test [20], which is a non-parametric test that makes no assumption about underlying data distribution, hence, raises the bar for significance for both normally and non-normally distributed data.…”

A Data Extraction Algorithm from Open Source Software Project Repositories for Building Duration Estimation Models: Case Study of Github

“…MacDonell et al [12] provided a transparent and consistent means of collection and evaluation of data that could lead to the use of higher quality data in software engineering experiments. They assessed the quality of 13 empirical software engineering datasets with the aim of benchmarking them against the data quality taxonomy proposed by Bosu and MacDonell [13] based on three groups: accuracy, relevance and provenance.…”