BACKGROUND -self evidently empirical analyses rely upon the quality of their data. Likewise replications rely upon accurate reporting and using the same rather than similar versions of data sets. In recent years there has been much interest in using machine learners to classify software modules into defectprone and not defect-prone categories. The publicly available NASA datasets have been extensively used as part of this research.OBJECTIVE -this short note investigates the extent to which published analyses based on the NASA defect data sets are meaningful and comparable. METHOD -we analyse the five studies published in IEEE Transactions on Software Engineering since2007 that have utilised these data sets and compare the two versions of the data sets currently in use.RESULTS -we find important differences between the two versions of the data sets, implausible values in one data set and generally insufficient detail documented on data set pre-processing.CONCLUSIONS -it is recommended that researchers (i) indicate the provenance of the data sets they use (ii) report any pre-processing in sufficient detail to enable meaningful replication and (iii) invest effort in understanding the data prior to applying machine learners.

No abstract

No abstract

No abstract