Building Better Quality Predictors Using "$ε$-Dominance"

Wei, Fu; Chen, Di; Agrawal, Amritanshu

doi:10.48550/arxiv.1803.04608

Cited by 3 publications

(8 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…-Another approach, is to find shortcuts around the optimization process. For recent work on that, which we would characterize as highly speculative, see [34].…”

Section: Research Directionmentioning

confidence: 99%

Better software analytics via “DUO”: Data mining algorithms using/used-by optimizers

et al. 2020

Self Cite

View full text Add to dashboard Cite

This paper claims that a new field of empirical software engineering research and practice is emerging: data mining using/used-by optimizers for empirical studies, or DUO. For example, data miners can generate the models that are explored by optimizers. Also, optimizers can advise how to best adjust the control parameters of a data miner. This combined approach acts like an agent leaning over the shoulder of an analyst that advises "ask this question next" or "ignore that problem, it is not relevant to your goals". Further, those agents can help us build "better" predictive models, where "better" can be either greater predictive accuracy, or faster modeling time (which, in turn, enables the exploration of a wider range of options). We also caution that the era of papers that just use data miners is coming to an end. Results obtained from an unoptimized data miner can be quickly refuted, just by applying an optimizer to produce a different (and better performing) model. Our conclusion, hence, is that for software analytics it is possible, useful and necessary to combine data mining and optimization using DUO.

show abstract

“…-Another approach, is to find shortcuts around the optimization process. For recent work on that, which we would characterize as highly speculative, see [34].…”

Section: Research Directionmentioning

confidence: 99%

Better software analytics via “DUO”: Data mining algorithms using/used-by optimizers

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…This standard of picking four different levels of performance between various data miners was also adapted for other recent empirical SE studies [2,8,14,21].…”

Section: Data Minersmentioning

confidence: 99%

“…Knowing that, is the insignificant difference in performance due to the nature of defect prediction problem itself or the traditional approach of exploring the tuning input space to the problem? For instance, Fu, Chen, and Agrawal [3,8,14] had applied simple method designed by psychological principles, Fast and Frugal Trees [35], that focusing on exploring the output space as binary tree with depth d = 4 instead of exploring the input space. This backward approach offers better or similar performance (for most cases) but with much less trade-off in term of time, processing power, and result's human-readability.…”

Section: Hyperparameter Tuningmentioning

confidence: 99%

“…Sampling Bias threatens any classification experiment, i.e.,what proves to be important here may be insignificant there. For e.g., We applied ten widely used open source JAVA software project data from SEACRAFT as the subject in various case studies by various researchers [8,14,15], i.e., our results are not more biased than many other studies in this arena. However, the datasets were supplied by one individual.…”

Section: Sampling Biasmentioning

confidence: 99%

See 1 more Smart Citation

Is one hyperparameter optimizer enough?

Nair

2018

Proceedings of the 4th ACM SIGSOFT International Workshop on Software Analytics

View full text Add to dashboard Cite

Hyperparameter tuning is the black art of automatically finding a good combination of control parameters for a data miner. While widely applied in empirical Software Engineering, there has not been much discussion on which hyperparameter tuner is best for software analytics. To address this gap in the literature, this paper applied a range of hyperparameter optimizers (grid search, random search, differential evolution, and Bayesian optimization) to a defect prediction problem. Surprisingly, no hyperparameter optimizer was observed to be "best" and, for one of the two evaluation measures studied here (F-measure), hyperparameter optimization, in 50% of cases, was no better than using default configurations.We conclude that hyperparameter optimization is more nuanced than previously believed. While such optimization can certainly lead to large improvements in the performance of classifiers used in software analytics, it remains to be seen which specific optimizers should be applied to a new dataset. CCS CONCEPTS• Software and its engineering → Search-based software engineering;1 These measures were used since they reference multiple target classes and optimizing for goals based on single targets leads to (e.g.) high recalls and false alarms 2 Most of the previous studies investigated just on project version [2,15].

show abstract

“…We used the similar implementation of FFT as offered by Fu and Chen et al [29], [51]. An FFT of depth d has a choice of two "exit policies" at each level: the existing branch can select for the negation of the target, i.e., non-severe, (denoted "0") or the target (denoted "1"), i.e., severe.…”

Section: E How Are Ffts Generated?mentioning

confidence: 99%

Can You Explain That, Better? Comprehensible Text Analytics for SE Applications

Agrawal,

Tu,

Menzies

2018

Preprint

Self Cite

View full text Add to dashboard Cite

Text mining methods are used for a wide range of Software Engineering (SE) tasks. The biggest challenge of text mining is high dimensional data, i.e., a corpus of documents can contain 10 4 to 10 6 unique words. To address this complexity, some very convoluted text mining methods have been applied. Is that complexity necessary? Are there simpler ways to quickly generate models that perform as well as the more convoluted methods and also be human-readable?To answer these questions, we explore a combination of LDA (Latent Dirichlet Allocation) and FFTs (Fast and Frugal Trees) to classify NASA software bug reports from six different projects. Designed using principles from psychological science, FFTs return very small models that are human-comprehensible. When compared to the commonly used text mining method and a recent state-of-the-art-system (search-based SE method that automatically tune the control parameters of LDA), these FFT models are very small (a binary tree of depth d = 4 that references only 4 topics) and hence easy to understand. They were also faster to generate and produced similar or better severity predictions.Hence we can conclude that, at least for datasets explored here, convoluted text mining models can be deprecated in favor of simpler method such as LDA+FFTs. At the very least, we recommend LDA+FFTs (a) when humans need to read, understand, and audit a model or (b) as an initial baseline method for the SE researchers exploring text artifacts from software projects.

show abstract

Building Better Quality Predictors Using "$ε$-Dominance"

Cited by 3 publications

References 39 publications

Better software analytics via “DUO”: Data mining algorithms using/used-by optimizers

Better software analytics via “DUO”: Data mining algorithms using/used-by optimizers

Is one hyperparameter optimizer enough?

Can You Explain That, Better? Comprehensible Text Analytics for SE Applications

Contact Info

Product

Resources

About