AVATAR - Machine Learning Pipeline Evaluation Using Surrogate Model

Nguyen, Tien-Dung; Maszczyk, Tomasz; Musiał, Katarzyna; Zöller, Marc-André; Gabrys, Bogdan

doi:10.1007/978-3-030-44584-3_28

Cited by 11 publications

(9 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This study is a significant extension of our previous work(Nguyen et al, 2020), a conference paper presented at the Eighteenth International Symposium on Intelligent Data Analysis…”

mentioning

confidence: 72%

AutoWeka4MCPS-AVATAR: Accelerating Automated Machine Learning Pipeline Composition and Optimisation

Nguyen¹,

Gabrys²,

Musiał³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Automated machine learning pipeline (ML) composition and optimisation aim at automating the process of finding the most promising ML pipelines within allocated resources (i.e., time, CPU and memory). Existing methods, such as Bayesian-based and genetic-based optimisation, which are implemented in Auto-Weka, Auto-sklearn and TPOT, evaluate pipelines by executing them. Therefore, the pipeline composition and optimisation of these methods frequently require a tremendous amount of time that prevents them from exploring complex pipelines to find better predictive models. To further explore this research challenge, we have conducted experiments showing that many of the generated pipelines are invalid in the first place, and attempting to execute them is a waste of time and resources.To address this issue, we propose a novel method to evaluate the validity of ML pipelines, without their execution, using a surrogate model (AVATAR). The AVATAR generates a knowledge base by automatically learning the capabilities and effects of ML algorithms on datasets' characteristics. This knowledge base is used for a simplified mapping from an original ML pipeline to a surrogate model which is a Petri net based pipeline. Instead of executing the original ML pipeline to evaluate its validity, the AVATAR evaluates its surrogate model constructed by capabilities and effects of the ML pipeline components and input/output simplified mappings. Evaluating this surrogate model is less resource-intensive than the execution of the original pipeline. As a result, the AVATAR enables the pipeline composition and optimisation methods to evaluate more pipelines by quickly rejecting invalid pipelines. We integrate the AVATAR into the sequential model-based algorithm configuration (SMAC). Our experiments show that when SMAC employs AVATAR, it finds better solutions than on its own. This is down to the fact that the AVATAR can evaluate more pipelines within the same time budget and allocated resources.

show abstract

“…This study is a significant extension of our previous work(Nguyen et al, 2020), a conference paper presented at the Eighteenth International Symposium on Intelligent Data Analysis…”

mentioning

confidence: 72%

AutoWeka4MCPS-AVATAR: Accelerating Automated Machine Learning Pipeline Composition and Optimisation

Nguyen¹,

Gabrys²,

Musiał³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The algorithms in a pipeline may be incompatible with each other due to implicit requirements of the used algorithms on the input data. Selecting and fitting all algorithms at once may lead to wasted optimization time as incompatibilities are only detected during fitting [16]. 3.…”

Section: Hyperparameter Optimizationmentioning

confidence: 99%

Incremental Search Space Construction for Machine Learning Pipeline Synthesis

Zöller

Nguyen

Huber

2021

Advances in Intelligent Data Analysis XIX

Self Cite

View full text Add to dashboard Cite

Automated machine learning (AutoML) aims for constructing machine learning (ML) pipelines automatically. Many studies have investigated efficient methods for algorithm selection and hyperparameter optimization. However, methods for ML pipeline synthesis and optimization considering the impact of complex pipeline structures containing multiple preprocessing and classification algorithms have not been studied thoroughly. In this paper, we propose a data-centric approach based on meta-features for pipeline construction and hyperparameter optimization inspired by human behavior. By expanding the pipeline search space incrementally in combination with meta-features of intermediate data sets, we are able to prune the pipeline structure search space efficiently. Consequently, flexible and data set specific ML pipelines can be constructed. We prove the effectiveness and competitiveness of our approach on 28 data sets used in well-established AutoML benchmarks in comparison with state-of-the-art AutoML frameworks.

show abstract

“…where the AutoML solution search process is significantly boosted by AVATAR [3], identifying and ignoring invalidly composed ML pipelines before they can waste evaluation time.…”

Section: A Experimental Settingsmentioning

confidence: 99%

Exploring Opportunistic Meta-knowledge to Reduce Search Spaces for Automated Machine Learning

Nguyen¹,

Kedziora²,

Musiał³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Machine learning (ML) pipeline composition and optimisation have been studied to seek multi-stage ML models, i.e. preprocessor-inclusive, that are both valid and wellperforming. These processes typically require the design and traversal of complex configuration spaces consisting of not just individual ML components and their hyperparameters, but also higher-level pipeline structures that link these components together. Optimisation efficiency and resulting ML-model accuracy both suffer if this pipeline search space is unwieldy and excessively large; it becomes an appealing notion to avoid costly evaluations of poorly performing ML components ahead of time. Accordingly, this paper investigates whether, based on previous experience, a pool of available classifiers/regressors can be preemptively culled ahead of initiating a pipeline composition/optimisation process for a new ML problem, i.e. dataset. The previous experience comes in the form of classifier/regressor accuracy rankings derived, with loose assumptions, from a substantial but non-exhaustive number of pipeline evaluations; this metaknowledge is considered 'opportunistic'. Numerous experiments with the AutoWeka4MCPS package, including ones leveraging similarities between datasets via the relative landmarking method, show that, despite its seeming unreliability, opportunistic meta-knowledge can improve ML outcomes. However, results also indicate that the culling of classifiers/regressors should not be too severe either. In effect, it is better to search through a 'top tier' of recommended predictors than to pin hopes onto one previously supreme performer.

show abstract

AVATAR - Machine Learning Pipeline Evaluation Using Surrogate Model

Cited by 11 publications

References 14 publications

AutoWeka4MCPS-AVATAR: Accelerating Automated Machine Learning Pipeline Composition and Optimisation

AutoWeka4MCPS-AVATAR: Accelerating Automated Machine Learning Pipeline Composition and Optimisation

Incremental Search Space Construction for Machine Learning Pipeline Synthesis

Exploring Opportunistic Meta-knowledge to Reduce Search Spaces for Automated Machine Learning

Contact Info

Product

Resources

About