Development Effort Estimation in Free/Open Source Software from Activity in Version Control Systems

Robles, Gregório; Capiluppi, Andrea; González-Barahona, Jesús M.; Lundell, Björn; Gamalielsson, Jonas

doi:10.48550/arxiv.2203.09898

Cited by 1 publication

(1 citation statement)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, the COCOMO model from Boehm et al [18] generated estimates of total project effort (measured in terms of 150-hour work-months) that were within 30% of actual, 71% of the time. Robbles et al [51] report that for a large class of OSS projects, developers will spend much of their time working full time on a single project. Hence, for those projects, it is possible to estimate the total effort that will be spent on a project, early in its development.…”

Section: From Effort To Health Estimationmentioning

confidence: 99%

Optimizing Predictions for Very Small Data Sets: a case study on Open-Source Project Health Prediction

Lustosa¹

2023

Preprint

View full text Add to dashboard Cite

When learning from very small data sets, the resulting models can make many mistakes. For example, consider learning predictors for open source project health. The training data for this task may be very small (e.g. five years of data, collected every month means just 60 rows of training data). Using this data, prior work had unacceptably large errors in their learned predictors.We show that these high errors rates can be tamed by better configuration of the control parameters of the machine learners. For example, we present here a landscape analytics method (called SNEAK) that (a) clusters the data to find the general landscape of the hyperparameters; then (b) explores a few representatives from each part of that landscape.SNEAK is both faster and and more effective than prior state-of-the-art hyperparameter optimization algorithms (FLASH, HYPEROPT, OPTUNA, and differential evolution). For example, SNEAK found its configurations after assessing 10 2 options, while other tools needed to assess 350 to 5000 (for OPTUNA, HYPEROPT) options. More importantly, the configurations found by SNEAK had far less error that other methods. For example, for project health indicators such as 𝐶= number of commits; 𝐼 =number of closed issues, and 𝑅=number of closed pull requests, SNEAK's 12 month prediction errors are {I=0%, R=33% C=47%} while other methods have far larger errors of {I=61%,R=119% C=149%}. We conjecture that SNEAK works so well since it finds the most informative regions of the hyperparameters, then jumps to those regions. Other methods (that do not reflect over the landscape) can waste time exploring less informative options.From this, we make the following conclusions. Firstly, for predicting open source project health, we recommend landscape analytics (e.g. SNEAK). Secondly, and more generally, when learning from very small data sets, using hyperparameter optimization (e.g. SNEAK) to select learning control parameters. Due to its speed and implementation simplicity, we suggest SNEAK might also be useful in other "data-light" SE domains.To assist other researchers in repeating, improving, or even refuting our results, all our scripts and data are available on GitHub at https://github.com/zxcv123456qwe/niSneak.

show abstract

Section: From Effort To Health Estimationmentioning

confidence: 99%

Optimizing Predictions for Very Small Data Sets: a case study on Open-Source Project Health Prediction

Lustosa¹

2023

Preprint

View full text Add to dashboard Cite

show abstract

Development Effort Estimation in Free/Open Source Software from Activity in Version Control Systems

Cited by 1 publication

References 44 publications

Optimizing Predictions for Very Small Data Sets: a case study on Open-Source Project Health Prediction

Optimizing Predictions for Very Small Data Sets: a case study on Open-Source Project Health Prediction

Contact Info

Product

Resources

About