Data complexity meta-features for regression problems

Lorena, Ana Carolina; Maciel, Aron I.; Miranda, Péricles; Costa, Ivan G.; Prudêncio, Ricardo B. C.

doi:10.1007/s10994-017-5681-1

Cited by 51 publications

(36 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…autoBagging ranks Bagging workflows including four different Bagging hyperparameters, using an XGBoost-based ranker, trained on 140 OpenML datasets and 146 meta-features. Lorena et al (2018) recommend SVM configurations for regression problems using a kNN meta-model and a new set of meta-features based on data complexity.…”

Section: Rankingmentioning

confidence: 99%

Meta-Learning

Vanschoren¹

2019

The Springer Series on Challenges in Machine Learning

321

166

View full text Add to dashboard Cite

Meta-learning, or learning to learn, is the science of systematically observing how different machine learning approaches perform on a wide range of learning tasks, and then learning from this experience, or meta-data, to learn new tasks much faster than otherwise possible. Not only does this dramatically speed up and improve the design of machine learning pipelines or neural architectures, it also allows us to replace hand-engineered algorithms with novel approaches learned in a data-driven way. In this chapter, we provide an overview of the state of the art in this fascinating and continuously evolving field.

show abstract

Section: Rankingmentioning

confidence: 99%

Meta-Learning

Vanschoren¹

2019

The Springer Series on Challenges in Machine Learning

321

166

View full text Add to dashboard Cite

show abstract

“…al. [25] proposed a set of complexity meta-features for regression problems. One of the case studies evaluated was the SVM HP tuning problem.…”

Section: Recommendation Of Hp Settingsmentioning

confidence: 99%

“…Our experimental results show that using meta-features from different categories have improved the predictive performance of the meta-learners for different setups. The most important meta-features were "SM.classes min" and "LM.stump sd" 25 . In [28], which only used meta-features from simple and data complexity sets, "SM.classes max" and "SM.attributes" were reported as the most important metafeatures.…”

Section: Linking Findings With the Literaturementioning

confidence: 99%

See 1 more Smart Citation

A meta-learning recommender system for hyperparameter tuning: Predicting when tuning improves SVM classifiers

Mantovani

Rossi

Alcobaça

et al. 2019

Information Sciences

View full text Add to dashboard Cite

For many machine learning algorithms, predictive performance is critically affected by the hyperparameter values used to train them. However, tuning these hyperparameters can come at a high computational cost, especially on larger datasets, while the tuned settings do not always significantly outperform the default values. This paper proposes a recommender system based on meta-learning to identify exactly when it is better to use default values and when to tune hyperparameters for each new dataset. Besides, an in-depth analysis is performed to understand what they take into account for their decisions, providing useful insights.An extensive analysis of different categories of meta-features, meta-learners, and setups across 156 datasets is performed. Results show that it is possible to accurately predict when tuning will significantly improve the performance of the induced models. The proposed system reduces the time spent on optimization processes, without reducing the predictive performance of the induced models (when compared with the ones obtained using tuned hyperparameters). We also explain the decision-making process of the meta-learners in terms of linear separability-based hypotheses. Although this analysis is focused on the tuning of Support Vector Machines, it can also be applied to other algorithms, as shown in experiments performed with decision trees.

show abstract

“…In the end, a kNN meta-model was applied to recommend the best HP setting for new unseen datasets. Lorena et al (2018) proposed a set of new complexity meta-features for regression problems. One of the case studies evaluated was the SVM HP tuning problem.…”

Section: Recommendation Of Hp Settingsmentioning

confidence: 99%

Use of meta-learning for hypeparameter tuning of classification problems

Mantovani¹

View full text Add to dashboard Cite

Machine learning solutions have been successfully used to solve many simple and complex problems. However, their development process still relies on human experts to perform tasks such as data preprocessing, feature engineering and model selection. As the complexity of these tasks increases, so does the demand for automated solutions, namely Automated Machine Learning (AutoML). Most algorithms employed in these systems have hyperparameters whose configuration may directly affect their predictive performance. Therefore, hyperparameter tuning is a recurring task in AutoML systems. This thesis investigated how to efficiently automate hyperparameter tuning by means of Meta-learning. To this end, large-scale experiments were performed tuning the hyperparameters of different classification algorithms, and an enhanced experimental methodology was adopted throughout the thesis to explore and learn the hyperparameter profiles for different classification algorithms. The results also showed that in many cases the default hyperparameter settings induced models that are on par with those obtained by tuning. Hence, a new Meta-learning recommender system was proposed to identify when it is better to use default values and when to tune classification algorithms for each new dataset. The proposed system is capable of generalizing several learning processes into a single modular framework, along with the possibility of assigning different algorithms. Furthermore, a descriptive analysis of model predictions is used to identify which data characteristics affect the necessity for tuning in each one of the algorithms investigated in the thesis. Experimental results also demonstrated that the proposed recommender system reduced the time spent on optimization processes, without reducing the predictive performance of the induced models. Depending on the target algorithm, the Meta-learning recommender system can statistically outperform the baselines. The significance of these results opens a number of new avenues for future work.

show abstract

Data complexity meta-features for regression problems

Cited by 51 publications

References 19 publications

Meta-Learning

Meta-Learning

A meta-learning recommender system for hyperparameter tuning: Predicting when tuning improves SVM classifiers

Use of meta-learning for hypeparameter tuning of classification problems

Contact Info

Product

Resources

About