Evaluating Pred(p) and standardized accuracy criteria in software development effort estimation

Abstract

Software development effort estimation (SDEE) plays a primary role in software project management. But choosing the appropriate SDEE technique remains elusive for many project managers and researchers. Moreover, the choice of a reliable estimation accuracy measure is crucial because SDEE techniques behave differently given different accuracy measures. The most widely used accuracy measures in SDEE are those based on magnitude of relative error (MRE) such as mean/median MRE (MMRE/MedMRE) and prediction at level p (Pred(p)), which counts the number of observations where an SDEE technique gave MREs lower than p. However, MRE has proven to be an unreliable accuracy measure, favoring SDEE techniques that underestimate. Consequently, an unbiased measure called standardized accuracy (SA) has been proposed. This paper deals with the Pred(p) and SA measures. We investigate (1) the consistency of Pred(p) and SA as accuracy measures and SDEE technique selectors, and (2) the relationship between Pred(p) and SA. The results suggest that Pred(p) is less biased towards underestimates and generally selects the same best technique as SA. Moreover, SA and Pred(p) measure different aspects of technique performance, and SA may be used as a predictor of Pred(p) by means of the 3 association rules.

Analysis of cluster center initialization of 2FA‐kprototypes analogy‐based software effort estimation

“…In spite of the widespread use of MMRE among researchers, it has been criticized for being biased toward underestimation making it an untrustworthy and unreliable accuracy measure. With regard to Pred(p), the study carried out by Idri et al showed that Pred(p) is less biased to underestimation and can be used as a reliable accuracy measure. Therefore, Pred(p) was retained in this study whereas MMRE was discarded.…”

Evaluating filter fuzzy analogy homogenous ensembles for software development effort estimation

“…This section describes the measures used to assess the performance of single and ensemble FA estimation techniques from 2 perspectives: Standardized accuracy (SA) and effect size assess the reasonability of a given estimation technique, ie, they check whether the estimation technique is actually predicting or guessing with respect to a baseline model; The other 5 performance criteria evaluate the accuracy of the estimation technique. …”

“…The Borda count method was used to draw the final ranking by combining the ranks provided by the 5 performance measures. The rationale for using many performance measures is that prior studies showed that selection of the best estimation technique depends on which performance indicator was used since relying on only 1 criterion may lead to biased conclusions . In other words, each technique can be ranked differently according to different accuracy criteria, which can lead to contradictory results (ie, 1 criterion selects model A as the best, whereas another selects model B) .…”

“…In other words, each technique can be ranked differently according to different accuracy criteria, which can lead to contradictory results (ie, 1 criterion selects model A as the best, whereas another selects model B) . For example, Idri et al showed that not all the best techniques according to SA were also the best with regard to Pred (p). Build and evaluate filter fuzzy analogy ensembles (RQ2) …”

“…These performance measures are unbiased and less impacted by an asymmetry assumption. The reader is reminded that the well‐known MMRE criterion was not adopted in this study as it has been criticized by many SDEE researchers . Another threat pointed that some results indicate that a technique outperforms other may not be really significant (eg, Albrecht for single FA techniques—Figure A, China dataset for best single and ensemble techniques—Figure B).…”