Learning From Mistakes: Machine Learning Enhanced Human Expert Effort Estimates

Abstract

In this paper, we introduce a novel approach to predictive modeling for software engineering, named Learning From Mistakes (LFM). The core idea underlying our proposal is to automatically learn from past estimation errors made by human experts, in order to predict the characteristics of their future misestimates, therefore resulting in improved future estimates. We show the feasibility of LFM by investigating whether it is possible to predict the type, severity and magnitude of errors made by human experts when estimating the development effort of software projects, and whether it is possible to use these predictions to enhance future estimations. To this end we conduct a thorough empirical study investigating 402 maintenance and new development industrial software projects. The results of our study reveal that the type, severity and magnitude of errors are all, indeed, predictable. Moreover, we find that by exploiting these predictions, we can obtain significantly better estimates than those provided by random guessing, human experts and traditional machine learners in 31 out of the 36 cases considered (86%), with large and very large effect sizes in the majority of these cases (81%). This empirical evidence opens the door to the development of techniques that use the power of machine learning, coupled with the observation that human errors are predictable, to support engineers in estimation tasks rather than replacing them with machine-provided estimates.

Multi-Objective Software Effort Estimation: A Replication Study

“…Meanwhile, estimating the most realistic amount of effort in the early stage of software development is difficult since the information available at that stage is usually incomplete and uncertain. Although construction of formal software effort estimation models started in the very early times of the industrialization of software production, expert judgement still remains the dominant strategy for effort prediction in practice where the accuracy of the estimate is sensitive to the practitioner's expertise and thus prone to bias [53], [87]. Early work to build an estimation technique tried to find a set of factors related to the software size and cost by using regression analysis [9].…”

“…Construct validity deals with the degree to which the predictor and response variable measures what they suppose to be measuring [71]. To mitigate possible threats arising from the predictors and target variables used to build the prediction models we used five publicly available datasets which were collected from real-world projects and contain reliable measures of software effort and size, such as Function Point, which are still in use in industrial settings and widely used in research studies [2], [27], [87]. Moreover, this data has been widely used in previous empirical studies to validate estimation models [10], [47], [48], [87].…”

“…To mitigate possible threats arising from the predictors and target variables used to build the prediction models we used five publicly available datasets which were collected from real-world projects and contain reliable measures of software effort and size, such as Function Point, which are still in use in industrial settings and widely used in research studies [2], [27], [87]. Moreover, this data has been widely used in previous empirical studies to validate estimation models [10], [47], [48], [87]. Further, to mitigate possible threats in measuring the accuracy performance of our models we used robust evaluation measures such as the Mean Absolute Error and the Standard Accuracy, which are widely used and recommended in previous work [64], [88], [95].…”

Deep Learning for Agile Effort Estimation Have We Solved the Problem Yet?

“…Choetkiertikul et al [5] also collected eight open-source projects stored in six open-source repositories, aiming at benchmarking Deep-SE against Porru's TF/IDF-SE approach using a common dataset. 7 In total, the Porru dataset, as collected by Choetkiertikul et al [5], [15], contains 4,904 issues. Among these eight projects, six are common with the Choe dataset (i.e., TIMOB, TISTUD, APSTUD, MESOS, MULE, and XD), although they contain a different subset of issues as Porru et al applied a set of more restrictive filtering criteria than those used by Choetkiertikul et al in building the Choet dataset [5].…”

“…Story Point (SP) is commonly used to measure the effort needed to implement a user story [2], [4] and agile teams mainly rely on expert-based estimation [1], [5]. However, similar to traditional software project effort estimation [6], [7], task-level effort estimation is not immune to the expert's subjective assessment [4]. Subjective assessment may not only lead to inaccurate estimations but also, and more importantly to an agile team, it may introduce inconsistency in estimates throughout different sprints.…”