Exact Mean Absolute Error of Baseline Predictor, MARP0

Langdon, William B.; Dolado, José Javier; Sarro, Federica; Harman, Mark

doi:10.1016/j.infsof.2016.01.003

Cited by 76 publications

(40 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most studies evaluate only a limited number of modeling techniques on a dataset, which limits the generalization of results. In addition, the results of different studies are difficult to compare due to their different empirical setups, data preprocessing, and dataset characteristics (Shepperd and MacDonell 2012;Langdon et al 2016). Therefore, the issue of which modeling technique to use for software effort estimation remains an open research question (Dejaeger et al 2012).…”

Section: Related Workmentioning

confidence: 99%

“…They concluded that different learning machines have different sensitivity to their parameter settings. Finally, Langdon et al 2016) proposed a measure based on a random guessing framework to compare methods for software estimation. Harman and Jones (2001) stated that software engineering is ideal for the application of metaheuristic search techniques, such as genetic algorithms, simulated annealing and tabu search.…”

Section: Framework For Benchmarking Prediction Modelsmentioning

confidence: 99%

“…The prediction accuracy of the models was tested following the criteria applied in (Shepperd and MacDonell 2012;Dejaeger et al 2012;Langdon et al 2016;Seo et al 2013;Lavazza et al 2013;Lokan and Mendes 2006). The differences between the actual effort (e i ) and the predicted effort (ê i ) should be as small as possible because large deviations between ei andêi will have a significant impact on the software development costs (Dejaeger et al 2012).…”

Section: Evaluation Criteriamentioning

confidence: 99%

“…Mean of the absolute residuals (MAR) and the standardized accuracy (SA) are new measures recommended to compare the performance of prediction models (Shepperd and MacDonell 2012;Langdon et al 2016). MAR is unbiased (towards over or underestimation) since it is not based on ratios, but it is hard to interpret and comparisons cannot be made across data sets since the residuals are not standardized.…”

Section: Evaluation Criteriamentioning

confidence: 99%

See 3 more Smart Citations

A genetic algorithm based framework for software effort prediction

Murillo-Morera

Quesada-López

Castro-Herrera

et al. 2017

J Softw Eng Res Dev

View full text Add to dashboard Cite

Background: Several prediction models have been proposed in the literature using different techniques obtaining different results in different contexts. The need for accurate effort predictions for projects is one of the most critical and complex issues in the software industry. The automated selection and the combination of techniques in alternative ways could improve the overall accuracy of the prediction models. Objectives: In this study, we validate an automated genetic framework, and then conduct a sensitivity analysis across different genetic configurations. Following is the comparison of the framework with a baseline random guessing and an exhaustive framework. Lastly, we investigate the performance results of the best learning schemes. Methods: In total, six hundred learning schemes that include the combination of eight data preprocessors, five attribute selectors and fifteen modeling techniques represent our search space. The genetic framework, through the elitism technique, selects the best learning schemes automatically. The best learning scheme in this context means the combination of data preprocessing + attribute selection + learning algorithm with the highest coefficient correlation possible. The selected learning schemes are applied to eight datasets extracted from the ISBSG R12 Dataset. Results: The genetic framework performs as good as an exhaustive framework. The analysis of the standardized accuracy (SA) measure revealed that all best learning schemes selected by the genetic framework outperforms the baseline random guessing by 45-80%. The sensitivity analysis confirms the stability between different genetic configurations. Conclusions: The genetic framework is stable, performs better than a random guessing approach, and is as good as an exhaustive framework. Our results confirm previous ones in the field, simple regression techniques with transformations could perform as well as nonlinear techniques, and ensembles of learning machines techniques such as SMO, M5P or M5R could optimize effort predictions.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Framework For Benchmarking Prediction Modelsmentioning

confidence: 99%

Section: Evaluation Criteriamentioning

confidence: 99%

Section: Evaluation Criteriamentioning

confidence: 99%

See 2 more Smart Citations

A genetic algorithm based framework for software effort prediction

Murillo-Morera

Quesada-López

Castro-Herrera

et al. 2017

J Softw Eng Res Dev

View full text Add to dashboard Cite

show abstract

“…Similarly to previous work in e↵ort estimation (e.g. [8,19,28,29]), we employed two widely-used standardised measures, Mean Absolute Error (MAE) and the Standardized Accuracy (SA). They are defined as below.…”

Section: Experimental Settings and Measuresmentioning

confidence: 99%

Multi-objective search-based approach to estimate issue resolution time

Al-Zubaidi

Dam

Ghose

et al. 2017

Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering

View full text Add to dashboard Cite

Background: Resolving issues is central to modern agile software development where a software is developed and evolved incrementally through series of issue resolutions. An issue could represent a requirement for a new functionality, a report of a software bug or a description of a project task. Aims: Knowing how long an issue will be resolved is thus important to different stakeholders including end-users, bug reporters, bug triagers, developers and managers. This paper aims to propose a multi-objective search-based approach to estimate the time required for resolving an issue. Methods: Using genetic programming (a meta-heuristic optimization method), we iteratively generate candidate estimate models and search for the optimal model in estimating issue resolution time. The search is guided simultaneously by two objectives: maximizing the accuracy of the estimation model while minimizing its complexity. Results: Our evaluation on 8,260 issues from five large open source projects demonstrate that our approach significantly (p < 0.001) outperforms both the baselines and state-of-the-art techniques. Conclusions: Evolutionary search-based approaches offer an effective alternative to build estimation models for issue resolution time. Using multiple objectives, one for measuring the accuracy and the other for the complexity, helps produce accurate and simple estimation models. Resolving issues is central to modern agile software development where a software is developed and evolved incrementally through series of issue resolutions. An issue could represent a requirement for a new functionality, a report of a software bug or a description of a project task.Aims: Knowing how long an issue will be resolved is thus important to di↵erent stakeholders including end-users, bug reporters, bug triagers, developers and managers. This paper aims to propose a multi-objective search-based approach to estimate the time required for resolving an issue.Methods: Using genetic programming (a meta-heuristic optimization method), we iteratively generate candidate estimate models and search for the optimal model in estimating issue resolution time. The search is guided simultaneously by two objectives: maximizing the accuracy of the estimation model while minimizing its complexity.Results: Our evaluation on 8,260 issues from five large open source projects demonstrate that our approach significantly (p < 0.001) outperforms both the baselines and state-of-the-art techniques.Conclusions: Evolutionary search-based approaches o↵er an e↵ective alternative to build estimation models for issue resolution time. Using multiple objectives, one for measuring the accuracy and the other for the complexity, helps produce accurate and simple estimation models.

show abstract

The state‐of‐the‐art in software development effort estimation

Gautam

Singh

2018

J Software Evolu Process

View full text Add to dashboard Cite

The software developers and researchers have been facing difficulties regarding software development effort estimation (SDEE) since 1960s. Both overestimation and underestimation are problematic for future software development. The software engineering field is continuously adapting new technologies and development methodologies, so there is always a requirement to have an accurate SDEE method that can cater the needs of continually growing software industry. The major purpose of this state‐of‐the‐art review is to provide an additional insight of existing SDEE studies while considering five points of reference: techniques used to construct models, strengths and weaknesses of different models, availability of benchmark data sets, data set characteristics, generalization ability of models. We have performed a comprehensive review of SDEE studies published in the period 1981‐2016. We have defined a new scheme of categorizing existing SDEE models. We have found that a majority of available data sets do not include complete information of projects, which misleads the direction of research. To compare SDEE models, we recommend to use same data sets while focusing on specific aspects of accuracy as none of SDEE studies has yet been able to compare all the existing models over same data sets while considering same aspects of accuracy.

show abstract

Exact Mean Absolute Error of Baseline Predictor, MARP0

Cited by 76 publications

References 1 publication

A genetic algorithm based framework for software effort prediction

A genetic algorithm based framework for software effort prediction

Multi-objective search-based approach to estimate issue resolution time

The state‐of‐the‐art in software development effort estimation

Contact Info

Product

Resources

About