Hyper-Parameter Tuning of a Decision Tree Induction Algorithm

Mantovani, Rafael Gomes; Horváth, Tomáš; Cerri, Ricardo; Vanschoren, Joaquin; Carvalho, André C. P. L. F. de

doi:10.1109/bracis.2016.018

Cited by 81 publications

(73 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…These are summarized in Figure and they are interconnected, so modifying the value of one parameter will impact the most suitable value of other parameters. This is the case for most machine learning models, but is especially challenging for boosted trees because they are sensitive to these hyper‐parameters with a relatively small window of optimal hyper‐parameters outside of which the model under or over fits. Throughout this work, we used Hyperopt, a Python library for automatically searching the hyper‐parameter space and making optimal choices.…”

Section: Hyper‐parameter Search Parallelizationmentioning

confidence: 99%

See 1 more Smart Citation

Machine learning on Crays to optimize petrophysical workflows in oil and gas exploration

Brown

Roubickova

Lampaki

et al. 2020

Concurrency and Computation

View full text Add to dashboard Cite

Summary The oil and gas industry is awash with sub‐surface data, which is used to characterize the rock and fluid properties beneath the seabed. This drives commercial decision making and exploration, but the industry relies upon highly manual workflows when processing data. A question is whether this can be improved using machine learning, complementing the activities of petrophysicists searching for hydrocarbons. In this paper, we present work using supervised learning with the aim of decreasing the petrophysical interpretation time down from over 7 days to 7 minutes. We describe the use of mathematical models that have been trained using raw well log data, to complete each of the four stages of a petrophysical interpretation workflow, in addition to initial data cleaning. We explore how the predictions from these models compare against the interpretations of human petrophysicists, and numerous options and techniques that were used to optimize the models. The result of this work is the ability, for the first time, to use machine learning for the entire petrophysical workflow.

show abstract

Section: Hyper‐parameter Search Parallelizationmentioning

confidence: 99%

“…Most importantly, for this work, boosted trees handle missing data values, as the corresponding branching is down weighted. However, a major challenge is that the model is more difficult to tune and highly sensitive to the hyper‐parameters setup …”

Section: Introductionmentioning

confidence: 99%

Machine learning on Crays to optimize petrophysical workflows in oil and gas exploration

Brown

Roubickova

Lampaki

et al. 2020

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…In contrast, it is a rather new insight that HPO can be used to adapt general-purpose pipelines to specific application domains [30]. Nowadays, it is also widely acknowledged that tuned hyperparameters improve over the default setting provided by common machine learning libraries [100,116,130,149].…”

mentioning

confidence: 99%

Hyperparameter Optimization

Feurer¹,

Hutter²

2019

The Springer Series on Challenges in Machine Learning

837

496

View full text Add to dashboard Cite

Recent interest in complex and computationally expensive machine learning models with many hyperparameters, such as automated machine learning (AutoML) frameworks and deep neural networks, has resulted in a resurgence of research on hyperparameter optimization (HPO). In this chapter, we give an overview of the most prominent approaches for HPO. We first discuss blackbox function optimization methods based on model-free methods and Bayesian optimization. Since the high computational demand of many modern machine learning applications renders pure blackbox optimization extremely costly, we next focus on modern multi-fidelity methods that use (much) cheaper variants of the blackbox function to approximately assess the quality of hyperparameter settings. Lastly, we point to open problems and future research directions. 1.1 Introduction Every machine learning system has hyperparameters, and the most basic task in automated machine learning (AutoML) is to automatically set these hyperparameters to optimize performance. Especially recent deep neural networks crucially depend on a wide range of hyperparameter choices about the neural network's architecture, regularization, and optimization. Automated hyperparameter optimization (HPO) has several important use cases; it can • reduce the human effort necessary for applying machine learning. This is particularly important in the context of AutoML.

show abstract

“…Furthermore, in general, the complete set of meta-features provided the best results for most algorithms. The meta-datasets for the J48 algorithm were generated based on HP tuning results obtained from 102 datasets reported in [26] 22 The J48 hyperparameter space is also presented in Appendix C. 23 In this paper, the "hyperparameter profile" term refers to how sensitive an algorithm may be to the HP tuning task. missing in the chart.…”

Section: A Note On the Generalizationmentioning

confidence: 99%

A meta-learning recommender system for hyperparameter tuning: Predicting when tuning improves SVM classifiers

Mantovani

Rossi

Alcobaça

et al. 2019

Information Sciences

Self Cite

View full text Add to dashboard Cite

For many machine learning algorithms, predictive performance is critically affected by the hyperparameter values used to train them. However, tuning these hyperparameters can come at a high computational cost, especially on larger datasets, while the tuned settings do not always significantly outperform the default values. This paper proposes a recommender system based on meta-learning to identify exactly when it is better to use default values and when to tune hyperparameters for each new dataset. Besides, an in-depth analysis is performed to understand what they take into account for their decisions, providing useful insights.An extensive analysis of different categories of meta-features, meta-learners, and setups across 156 datasets is performed. Results show that it is possible to accurately predict when tuning will significantly improve the performance of the induced models. The proposed system reduces the time spent on optimization processes, without reducing the predictive performance of the induced models (when compared with the ones obtained using tuned hyperparameters). We also explain the decision-making process of the meta-learners in terms of linear separability-based hypotheses. Although this analysis is focused on the tuning of Support Vector Machines, it can also be applied to other algorithms, as shown in experiments performed with decision trees.

show abstract

Hyper-Parameter Tuning of a Decision Tree Induction Algorithm

Cited by 81 publications

References 22 publications

Machine learning on Crays to optimize petrophysical workflows in oil and gas exploration

Machine learning on Crays to optimize petrophysical workflows in oil and gas exploration

Hyperparameter Optimization

A meta-learning recommender system for hyperparameter tuning: Predicting when tuning improves SVM classifiers

Contact Info

Product

Resources

About