An empirical evaluation of outlier deletion methods for analogy-based cost estimation

Abstract

Background: Any software project dataset sometimes includes outliers which affect the accuracy of effort estimation. Outlier deletion methods are often used to eliminate them. However, there are few case studies which apply outlier deletion methods to analogy-based estimation, so it is not clear which method is more suitable for analogy-based estimation. Aim: Clarifying the effects of existing outlier deletion methods (Cook's distance based deletion, LTS based deletion, k-means based deletion, Mantel's correlation based deletion, and EID based deletion) and our method for analogy-based estimation. Method: In the experiment, outlier deletion methods were applied to three kinds of datasets (the ISBSG, Kitchenham, and Desharnais datasets), and their estimation accuracy evaluated based on BRE (Balanced Relative Error). Our method eliminates outliers from the neighborhoods of a target project when the effort is extremely different from other neighborhoods. Results: Deletion methods which are designed to apply to analogy-based estimation (i.e. Mantel's correlation based deletion, EID based deletion, and our method) showed stable performance. Especially, only our method showed over 10% improvement of the average BRE on two datasets. Conclusions: It is reasonable to apply deletion methods designed for analogy-based estimation, and more preferable to apply our method to analogybased estimation.

An empirical analysis of data preprocessing for machine learning-based software cost estimation

“…The combination of scaling scheme and missing-data treatment (MDT) is firstly analyzed; however, their impacts onto the ML method were not studied. Many studies propose one or more DP techniques to deal with a specific issue in SEC, such as data missingness [30][31][32], redundant or irrelevant features [13,28], or abnormal cases [33,34]. But they did not study the effectiveness of different DP techniques.…”

“…Table 1 also shows that many studies use combined DPs. For examples, there are 7 of totally 48 works combined only scaling and FS/CS [20,27,34,[52][53][54][55], and 7 of 48 works combined only MDTs and FS/CS [19,23,36,[56][57][58][59]. FS and CS have been considered as a necessary step for CBR in several studies [6,14,23,26,33,37,47,48,52,[55][56][57][60][61][62][63].…”

“…random search, hill climbing, simulated annealing, forward and backward sequential selection [60,84], genetic algorithm [28], and filters [81,82]. On contrary to the selection of useful cases, the abnormal cases detection [33,34] is also regarded as one type of CS. Compared to FS, the CS treatment is relatively infrequent but with a rising trend in the recent literature.…”

“…Table 3 shows the statistics of the numerical features in details. Table 3 Descriptive statistics of numeric features of Desharnais dataset [34]. Table 4 shows the statistics of the numerical features in details.…”

Fine-Grained Work Element Standardization for Project Effort Estimation

“…This step deals with two imposing problems in data collection, namely, missing value and outlier detection [22]. Missing value is a common phenomenon in project management where numerous small activities, assignments, and work products are too trivial, hard to identify or account for, or due to human errors.…”