A Comparison of AutoML Tools for Machine Learning, Deep Learning and XGBoost

Abstract

This paper presents a benchmark of supervised Automated Machine Learning (AutoML) tools. Firstly, we analyze the characteristics of eight recent open-source AutoML tools (Auto-Keras, Auto-PyTorch, Auto-Sklearn, AutoGluon, H2O AutoML, rminer, TPOT and TransmogrifAI) and describe twelve popular OpenML datasets that were used in the benchmark (divided into regression, binary and multi-class classification tasks). Then, we perform a comparison study with hundreds of computational experiments based on three scenarios: General Machine Learning (GML), Deep Learning (DL) and XGBoost (XGB). To select the best tool, we used a lexicographic approach, considering first the average prediction score for each task and then the computational effort. The best predictive results were achieved for GML, which were further compared with the best OpenML public results. Overall, the best GML AutoML tools obtained competitive results, outperforming the best OpenML models in five datasets. These results confirm the potential of the general-purpose AutoML tools to fully automate the Machine Learning (ML) algorithm selection and tuning.

Machine Learning Models for Slope Stability Classification of Circular Mode Failure: An Updated Database and Automated Machine Learning (AutoML) Approach

“…Various open-source platforms, such as AutoKeras, AutoPyTorch, AutoSklearn, AutoGluon, and H2O AutoML, have been developed to facilitate the adoption of AutoML [ 46 ]. Previous studies [ 47 , 48 ] have demonstrated the strong feature of H2O AutoML for processing large and complicated datasets by quickly searching the optimal model without the need for manual trial and error.…”

Can We Detect Malicious Behaviours in Encrypted DNS Tunnels Using Network Flow Entropy?

“…They evaluated all data using TPOT and showed that the larger the dataset the better the optimization of the classifiers works using TPOT. Ferreira et al carried out experiments based on three scenarios: General ML (GML), Deep Learning (DL) and XGBoost [27]. They performed a 10-fold cross-validation comparing eight open-source AutoML tools on 12 popular OpenML datasets.…”

Avaliação de Ferramentas de AutoML em Datasets de Detecção de Malwares Android

“…Diversos trabalhos recentes avaliam ou propõe ferramentas de AutoML para diferentes contexto e domínios [Nagarajah and Poravi, 2019, Ferreira et al, 2021, Bezrukavnikov and Linder, 2021, Truong et al, 2019, Karmaker et al, 2021. Há estudos recentes (e.g., [Nagarajah andPoravi, 2019, Karmaker et al, 2021]) que apresentam análises de dezenas de trabalhos de AutoML, observando as particularidades de diferentes ferramentas, como Auto-Weka, Hyperopt-Sklearn, TPOT, AutoCompet e PennAI, e apontando desafios na área, como interpretabilidade, transparência, níveis de personalizac ¸ão e recursos de depurac ¸ão.…”

“…Em [Ferreira et al, 2021], os autores apresentam um estudo comparativo entre aplicac ¸ões de AutoML quanto a machine learning, deep learning e XGBoost. As ferramentas Auto-Keras, Auto-PyTorch, Auto-Sklearn, AutoGluon, H2O AutoML, rminer, TPOT e TransmogrifAI foram avaliadas com relac ¸ão ao tempo de execuc ¸ão e desempenho de predic ¸ão em doze datasets, disponíveis na plataforma OpenML 7 , em diferentes domínios de aplicac ¸ão, como a área médica (diagnóstico de doenc ¸as cardíacas, concentrac ¸ão de plasma e diabetes), métodos contraceptivos e classificac ¸ão de risco de um cliente para concessão de crédito.…”

“…1. contemplarem todas as etapas do processo de AutoML, segundo as etapas descritas em [Nagarajah and Poravi, 2019]; 2. possuírem código-aberto; 3. serem atuais e incluírem mecanismos e métodos atualizados de machine learning; 4. serem utilizadas por outros autores em trabalhos recentes [Ferreira et al, 2021, Nagarajah and Poravi, 2019, Truong et al, 2019, Erickson et al, 2020]; e 5. serem parametrizáveis e funcionarem em linha de comando, permitindo a utilizac ¸ão de diferentes datasets e a automac ¸ão dos processos de testes e execuc ¸ão.…”