Hyperparameter Importance for Image Classification by Residual Neural Networks

Sharma, Abhinav; Rijn, Jan N. van; Hutter, Frank; Müller, Andreas

doi:10.1007/978-3-030-33778-0_10

Cited by 17 publications

(14 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Generally, the correlations between budgets are rather high as required for multi-fidelity optimization. As expected, the adjacent budget pairs (12,25) and (25, 50) exhibit a larger correlation than the more distant budget pair (12,50). Table 3 lists the mean correlation across all tasks.…”

Section: Rq3: Correlation Between Budgetssupporting

confidence: 77%

“…Last but not least, several papers address the problem of understanding the characteristics of AutoDL tasks [12], [25]. A typical finding for example is that the design space is over-engineered, leading to very simple optimization tasks where even random search can perform well.…”

Section: Related Workmentioning

confidence: 99%

“…As we show in our experiments, this does not apply to our tasks. In addition to study the characteristics of LCBench and also the full design space of Auto-PyTorch Tabular, we study hyperparameter importance from a global [25], [26], [27], but also from a local point of view [28], showing that both provide different insights.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL

Zimmer¹,

Lindauer²,

Hutter³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

While early AutoML frameworks focused on optimizing traditional ML pipelines and their hyperparameters, a recent trend in AutoML is to focus on neural architecture search. In this paper, we introduce Auto-PyTorch, which brings the best of these two worlds together by jointly and robustly optimizing the architecture of networks and the training hyperparameters to enable fully automated deep learning (AutoDL). Auto-PyTorch achieves state-of-the-art performance on several tabular benchmarks by combining multi-fidelity optimization with portfolio construction for warmstarting and ensembling of deep neural networks (DNNs) and common baselines for tabular data. To thoroughly study our assumptions on how to design such an AutoDL system, we additionally introduce a new benchmark on learning curves for DNNs, dubbed LCBench, and run extensive ablation studies of the full Auto-PyTorch on typical AutoML benchmarks, eventually showing that Auto-PyTorch performs better than several state-of-the-art competitors on average.

show abstract

Section: Rq3: Correlation Between Budgetssupporting

confidence: 77%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL

Zimmer¹,

Lindauer²,

Hutter³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…For instance, Probst et al (2019) are specifically interested in binary classification and hence use a subset of the "OpenML-100" with a binary target. Similarly, Sharma et al (2019) are interested in image classification, and therefore define a set of ten image datasets.…”

Section: Establishing Hyperparameter Importance Across Datasetsmentioning

confidence: 99%

Setting Up Configuration Spaces and Experiments

Brazdil

Rijn

Soares³

et al. 2022

Metalearning

Self Cite

View full text Add to dashboard Cite

SummaryThis chapter discusses the issues relative to so-called configuration spaces that need to be set up before initiating the search for a solution. It starts by introducing some basic concepts, such as discrete and continuous subspaces. Then it discusses certain criteria that help us to determine whether the given configuration space is (or is not) adequate for the tasks at hand. One important topic which is addressed here is hyperparameter importance, as it helps us to determine which hyperparameters have a high influence on the performance and should therefore be optimized. This chapter also discusses some methods for reducing the configuration space. This is important as it can speed up the process of finding the potentially best workflow for the new task. One problem that current systems face nowadays is that the number of alternatives in a given configuration space can be so large that it is virtually impossible to gather complete metadata. This chapter discusses the issue of whether the system can still function satisfactorily even when the metadata is incomplete. The final part of this chapter discusses some strategies that can be used for gathering metadata that originated in the area of multi-armed bandits, including, for instance, SoftMax, upper confidence bound (UCB) and pricing strategies.

show abstract

“…Hutter et al [2014a] quantified the importance of hyperparameters based on a random forest fitted on data generated by BO, for which the importance of both the main and the interaction effects of hyperparameters was calculated by a functional ANOVA approach. Similarly, Sharma et al [2019] quantified the hyperparameter importance of residual neural networks. These works highlight how useful it is to quantify the importance of hyperparameters.…”

Section: Background and Related Workmentioning

confidence: 99%

Explaining Hyperparameter Optimization via Partial Dependence Plots

Moosbauer¹,

Herbinger²,

Casalicchio³

et al. 2021

Preprint

View full text Add to dashboard Cite

Automated hyperparameter optimization (HPO) can support practitioners to obtain peak performance in machine learning models. However, there is often a lack of valuable insights into the effects of different hyperparameters on the final model performance. This lack of explainability makes it difficult to trust and understand the automated HPO process and its results. We suggest using interpretable machine learning (IML) to gain insights from the experimental data obtained during HPO with Bayesian optimization (BO). BO tends to focus on promising regions with potential high-performance configurations and thus induces a sampling bias. Hence, many IML techniques, such as the partial dependence plot (PDP), carry the risk of generating biased interpretations. By leveraging the posterior uncertainty of the BO surrogate model, we introduce a variant of the PDP with estimated confidence bands. We propose to partition the hyperparameter space to obtain more confident and reliable PDPs in relevant sub-regions. In an experimental study, we provide quantitative evidence for the increased quality of the PDPs within sub-regions.

show abstract

Hyperparameter Importance for Image Classification by Residual Neural Networks

Cited by 17 publications

References 14 publications

Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL

Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL

Setting Up Configuration Spaces and Experiments

Explaining Hyperparameter Optimization via Partial Dependence Plots

Contact Info

Product

Resources

About