Selecting Classification Algorithms with Active Testing

Leite, Rui; Brazdil, Pavel; Vanschoren, Joaquin

doi:10.1007/978-3-642-31537-4_10

Cited by 69 publications

(70 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another MTL approach, named Active Testing [29], [30], selects the algorithm and its hyper-parameters simultaneously. It uses the evaluations of all hyper-parameter settings tested on the new data set, and all evaluations on prior data sets, to select the best new candidate algorithm-parameter combination in the next iteration.…”

Section: Related Workmentioning

confidence: 99%

To tune or not to tune: Recommending when to adjust SVM hyper-parameters via meta-learning

Mantovani

Rossi

Vanschoren

et al. 2015

2015 International Joint Conference on Neural Networks (IJCNN)

Self Cite

View full text Add to dashboard Cite

Abstract-Many classification algorithms, such as Neural Net works and Support Vector Machines, have a range of hyper parameters that may strongly affect the predictive performance of the models induced by them. Hence, it is recommended to define the values of these hyper-parameters using optimization techniques. While these techniques usually converge to a good set of values, they typically have a high computational cost, because many candidate sets of values are evaluated during the optimization process. It is often not clear whether this will result in parameter settings that are significantly better than the default settings. When training time is limited, it may help to know when these parameters should definitely be tuned. In this study, we use meta-learning to predict when optimization techniques are expected to lead to models whose predictive performance is better than those obtained by using default parameter settings. Hence, we can choose to employ optimization techniques only when they are expected to improve performance, thus reducing the overall computational cost. We evaluate these meta-learning techniques on more than one hundred data sets. The experimental results show that it is possible to accurately predict when optimization techniques should be used instead of default values suggested by some machine learning libraries.

show abstract

Section: Related Workmentioning

confidence: 99%

To tune or not to tune: Recommending when to adjust SVM hyper-parameters via meta-learning

Mantovani

Rossi

Vanschoren

et al. 2015

2015 International Joint Conference on Neural Networks (IJCNN)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Dealing with the CASH optimization problem, we found [6] and [7]. Thornton et al [6] use recent innovations in Bayesian optimization to find the best parameter values for a classification algorithm.…”

Section: Related Workmentioning

confidence: 99%

“…Leite et al [7] propose a new technique for selecting classification algorithms called active testing. This technique selects the most useful algorithm using cross-validation testing on a tournament where, in each round of selection and test, it is chosen the algorithm that outperformed the algorithms that won the previous round.…”

Section: Related Workmentioning

confidence: 99%

“…We adopt k = 10 because it is a value that provides good estimates as established by several research papers [15], [7], [6], [10].…”

Section: A Cash Optimization Problemmentioning

confidence: 99%

“…On the one hand, such an abundance provides the researcher with proven techniques to mine accurate and general classification models out of raw data. On the other hand, the researcher is faced with the problem of choosing among tenths, or even hundreds, of sophisticated mining algorithms, each one having a set of (hyper)parameters that need to be adjust for the proper functioning of the algorithm [6], [7]. Considering that the medical researcher is not necessarily an expert in data mining, the right choice of algorithm and associated parameter values becomes a challenging task prone to error, or leading to sub-optimal selection based on intuitive appeal and/or reputation.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

EMiner: A Tool for Selecting Classification Algorithms and Optimal Parameters

Marques¹,

Coutinho²,

Borchartt³

et al. 2015

Polibits

View full text Add to dashboard Cite

Abstract-In this paper, Genetic Algorithm (GA) is used to search for combinations of learning algorithms and associated parameters with maximum accuracy. An important feature of the approach is that the GA initial population is formed by using parameter values gathered from ExpDB (a public database of data mining experiments). The proposed approach was implemented in a tool called EMiner, built on top of a grid based software infrastructure for developing collaborative applications in medicine and healthcare domains (ECADeG project). Experiments on 16 datasets from the UCI repository were performed. The results obtained have shown that the strategy of combining the data from ExpDB via GA is effective in finding classification models with good accuracy.

show abstract

Metalearning

Brazdil¹,

Vilalta²,

Giraud‐Carrier³

et al. 2016

Encyclopedia of Machine Learning and Data Mining

Self Cite

View full text Add to dashboard Cite

In the area machine learning / data mining many diverse algorithms are available nowadays and hence the selection of the most suitable algorithm may be a challenge. Tbhis is aggravated by the fact that many algorithms require that certain parameters be set. If a wrong algorithm and/or parameter configuration is selected, substandard results may be obtained. The topic of metalearning aims to facilitate this task. Met

show abstract

Selecting Classification Algorithms with Active Testing

Cited by 69 publications

References 15 publications

To tune or not to tune: Recommending when to adjust SVM hyper-parameters via meta-learning

To tune or not to tune: Recommending when to adjust SVM hyper-parameters via meta-learning

EMiner: A Tool for Selecting Classification Algorithms and Optimal Parameters

Metalearning

Contact Info

Product

Resources

About