OpenML: A Collaborative Science Platform

Rijn, Jan N. van; Bischl, Bernd; Torgo, Luı́s; Gao, Bo; Umaashankar, Venkatesh; Fischer, Simon; Winter, Patrick; Wiswedel, Bernd; Berthold, Michael R.; Vanschoren, Joaquin

doi:10.1007/978-3-642-40994-3_46

Cited by 46 publications

(28 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, we should include a significance test to define the meta-target, and include experiments with other classification algorithms instead of SVMs, such as decision trees and Deep Learning algorithms, which have a larger number of sensitive hyper parameters. Finally, we aim to make all our experiments avail able on OpenML [43], [44] for reproducibility and reusability.…”

Section: Co Nclusionsmentioning

confidence: 99%

To tune or not to tune: Recommending when to adjust SVM hyper-parameters via meta-learning

Mantovani

Rossi

Vanschoren

et al. 2015

2015 International Joint Conference on Neural Networks (IJCNN)

Self Cite

View full text Add to dashboard Cite

Abstract-Many classification algorithms, such as Neural Net works and Support Vector Machines, have a range of hyper parameters that may strongly affect the predictive performance of the models induced by them. Hence, it is recommended to define the values of these hyper-parameters using optimization techniques. While these techniques usually converge to a good set of values, they typically have a high computational cost, because many candidate sets of values are evaluated during the optimization process. It is often not clear whether this will result in parameter settings that are significantly better than the default settings. When training time is limited, it may help to know when these parameters should definitely be tuned. In this study, we use meta-learning to predict when optimization techniques are expected to lead to models whose predictive performance is better than those obtained by using default parameter settings. Hence, we can choose to employ optimization techniques only when they are expected to improve performance, thus reducing the overall computational cost. We evaluate these meta-learning techniques on more than one hundred data sets. The experimental results show that it is possible to accurately predict when optimization techniques should be used instead of default values suggested by some machine learning libraries.

show abstract

Section: Co Nclusionsmentioning

confidence: 99%

To tune or not to tune: Recommending when to adjust SVM hyper-parameters via meta-learning

Mantovani

Rossi

Vanschoren

et al. 2015

2015 International Joint Conference on Neural Networks (IJCNN)

Self Cite

View full text Add to dashboard Cite

show abstract

“…We have performed an extensive experiment on meta-learning on data streams, running a wide range of steam mining algorithms over a large number of data streams, and published all results online in OpenML [15], so that others can verify, reproduce and build upon these results. Containing more than 1,000 experiments on data streams, with extensive meta-information calculated over data (windows), this now forms a rich source of meta-learning experiments on data streams.…”

Section: Discussionmentioning

confidence: 99%

“…Another recent development is the concept of experiment databases [15,19], databases which contain detailed information about a large range of experiments. Experiment databases enable the reproduction of earlier results for verification and reusability purposes, and make much larger studies (covering more algorithms and parameter settings) feasible [19].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Algorithm Selection on Data Streams

Rijn

Holmes

Pfahringer

et al. 2014

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. We explore the possibilities of meta-learning on data streams, in particular algorithm selection. In a first experiment we calculate the characteristics of a small sample of a data stream, and try to predict which classifier performs best on the entire stream. This yields promising results and interesting patterns. In a second experiment, we build a meta-classifier that predicts, based on measurable data characteristics in a window of the data stream, the best classifier for the next window. The results show that this meta-algorithm is very competitive with state of the art ensembles, such as OzaBag, OzaBoost and Leveraged Bagging. The results of all experiments are made publicly available in an online experiment database, for the purpose of verifiability, reproducibility and generalizability.

show abstract

“…We also aim to build on, and make all our experiments available in OpenML [27], [28] for reproducibility and further study.…”

Section: A Comparison Of Tuning Strategiesmentioning

confidence: 99%