Is one hyperparameter optimizer enough?

Tu, Huy; Nair, Vivek

doi:10.1145/3278142.3278145

Cited by 13 publications

(5 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…learning on static code attributes such as C.K. and McGabe metrics) [1], [3], [18], [20], [33], [36], [44], [57], [64], [65], [72], [74], [80], [96] that are more granulated and high-dimensional.…”

Section: Future Workmentioning

confidence: 99%

Better Data Labelling With EMBLEM (and how that Impacts Defect Prediction)

2022

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

Standard automatic methods for recognizing problematic development commits can be greatly improved via the incremental application of human+artificial expertise. In this approach, called EMBLEM, an AI tool first explore the software development process to label commits that are most problematic. Humans then apply their expertise to check those labels (perhaps resulting in the AI updating the support vectors within their SVM learner). We recommend this human+AI partnership, for several reasons. When a new domain is encountered, EMBLEM can learn better ways to label which comments refer to real problems. Also, in studies with 9 open source software projects, labelling via EMBLEM's incremental application of human+AI is at least an order of magnitude cheaper than existing methods (≈ eight times). Further, EMBLEM is very effective. For the data sets explored here, EMBLEM better labelling methods significantly improved Popt20 and G-score performance in nearly all the projects studied here. TABLE 1This paper argues against using keywords like these as a method for labelling a commit as "buggy'.

show abstract

Section: Future Workmentioning

confidence: 99%

Better Data Labelling With EMBLEM (and how that Impacts Defect Prediction)

2022

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

show abstract

“…The problem with hyperparameter optimization is finding enough CPU. The cost of running a data miner through all those options is very high, requiring days to weeks to decades of CPU [56], [57], [58], [59], [62], [64]. For many years, we have addressed these long CPU times via cloudbased CPU farms.…”

Section: Introductionmentioning

confidence: 99%

How to “DODGE” Complex Software Analytics

Agrawal

Wei

Chen

et al. 2021

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Machine learning techniques applied to software engineering tasks can be improved by hyperparameter optimization, i.e., automatic tools that find good settings for a learner's control parameters. We show that such hyperparameter optimization can be unnecessarily slow, particularly when the optimizers waste time exploring "redundant tunings", i.e., pairs of tunings which lead to indistinguishable results. By ignoring redundant tunings, DODGE(E), a tuning tool, runs orders of magnitude faster, while also generating learners with more accurate predictions than seen in prior state-of-the-art approaches.

show abstract

“…-Some initial results suggest it may not be enough to just use AutoML [97] (in summary, given N hyperparameter optimizers, AutoML was "best" for a miniority of goals and datasets).…”

Section: Research Directionmentioning

confidence: 99%

“…-Some initial results suggest it may not be enough to just use AutoML [97] (in summary, given N hyperparameter optimizers, AutoML was "best" for a miniority of goals and datasets). -In terms of training ourselves on how easy it is to combine optimizers and data miners, there is no substitute for "rolling your own" implementation (at least once, then perhaps moving on to tools like AutoML).…”

Section: Research Directionmentioning

confidence: 99%

Better software analytics via “DUO”: Data mining algorithms using/used-by optimizers

et al. 2020

View full text Add to dashboard Cite

This paper claims that a new field of empirical software engineering research and practice is emerging: data mining using/used-by optimizers for empirical studies, or DUO. For example, data miners can generate the models that are explored by optimizers. Also, optimizers can advise how to best adjust the control parameters of a data miner. This combined approach acts like an agent leaning over the shoulder of an analyst that advises "ask this question next" or "ignore that problem, it is not relevant to your goals". Further, those agents can help us build "better" predictive models, where "better" can be either greater predictive accuracy, or faster modeling time (which, in turn, enables the exploration of a wider range of options). We also caution that the era of papers that just use data miners is coming to an end. Results obtained from an unoptimized data miner can be quickly refuted, just by applying an optimizer to produce a different (and better performing) model. Our conclusion, hence, is that for software analytics it is possible, useful and necessary to combine data mining and optimization using DUO.

show abstract

Is one hyperparameter optimizer enough?

Cited by 13 publications

References 41 publications

Better Data Labelling With EMBLEM (and how that Impacts Defect Prediction)

Better Data Labelling With EMBLEM (and how that Impacts Defect Prediction)

How to “DODGE” Complex Software Analytics

Better software analytics via “DUO”: Data mining algorithms using/used-by optimizers

Contact Info

Product

Resources

About