PACOH: Bayes-Optimal Meta-Learning with PAC-Guarantees

Rothfuss, Jonas; Fortuin, Vincent; Josifoski, Martin; Krause, Andreas

doi:10.48550/arxiv.2002.05551

Cited by 8 publications

(23 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the next section, we extend this framework to transfer meta-learning by leveraging the theoretical results in [7]. To this end, we first review non-parametric Bayesian learning via Gaussian Processes (GPs), and then we describe the problem of meta-learning the GP prior proposed in [8]. Notation: We use Roman fonts to indicate random variables, functions, and vectors; while the corresponding regular font denotes fixed realizations.…”

Section: Problem Settingmentioning

confidence: 99%

“…We will also require the evidence or marginal likelihood of the output labels, p θ (Y|X ) = p θ (t(X ) = t)p(Y|t(X ) = t)dt, (8) the log of which can be obtained in closed form for the Gaussian likelihood as [18]…”

Section: Gaussian Processes (Gps)mentioning

confidence: 99%

“…In GP, the hyperparameter vector θ ∈ Θ that describes the mean and kernel functions of the GP in ( 4) is fixed a priori, possibly using cross-validation. In contrast, the metalearning approach introduced in [8] -termed PACOH-GP -aims to automatically infer the hyperparameters θ of the GP prior by observing data from tasks with similar statistical properties [8]. Note that, the kernel function could be parametrized as…”

Section: Gaussian Processes (Gps)mentioning

confidence: 99%

“…We specifically focus on non-parametric Bayesian learning via Gaussian processes (GPs), and aim to meta-learn a GP prior to be used on meta-test tasks. WFEM generalizes the information meta-risk minimization (IMRM) for transfer meta-learning introduced by the theory in [7] to nonparametric base-learners, as well as the PACOH-GP based Bayesian meta-learning approach of [8].…”

Section: Introductionmentioning

confidence: 99%

“…In this context, both parametric methods such as Bayesian neural networks and non-parametric methods like GPs have both been successfully applied to real-world applications. The works in [8,16] apply meta-learning to optimize the mean and kernel functions of the GP prior via parametric functions -a method referred to as PACOH-GP. Our work extends PACOH-GP to transfer meta-learning.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Transfer Bayesian Meta-Learning Via Weighted Free Energy Minimization

Yunchuan

Jose

Simeone

2021

2021 IEEE 31st International Workshop on Machine Learning for Signal Processing (MLSP)

View full text Add to dashboard Cite

Meta-learning optimizes the hyperparameters of a training procedure, such as its initialization, kernel, or learning rate, based on data sampled from a number of auxiliary tasks.A key underlying assumption is that the auxiliary tasksknown as meta-training tasks -share the same generating distribution as the tasks to be encountered at deployment time -known as meta-test tasks. This may, however, not be the case when the test environment differ from the meta-training conditions. To address shifts in task generating distribution between meta-training and meta-testing phases, this paper introduces weighted free energy minimization (WFEM) for transfer meta-learning. We instantiate the proposed approach for non-parametric Bayesian regression and classification via Gaussian Processes (GPs). The method is validated on a toy sinusoidal regression problem, as well as on classification using miniImagenet and CUB data sets, through comparison with standard metalearning of GP priors as implemented by PACOH.

show abstract

Section: Problem Settingmentioning

confidence: 99%

Section: Gaussian Processes (Gps)mentioning

confidence: 99%

Section: Gaussian Processes (Gps)mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Transfer Bayesian Meta-Learning Via Weighted Free Energy Minimization

Yunchuan

Jose

Simeone

2021

2021 IEEE 31st International Workshop on Machine Learning for Signal Processing (MLSP)

View full text Add to dashboard Cite

show abstract

Transfer Meta-Learning: Information-Theoretic Bounds and Information Meta-Risk Minimization

Jose¹,

Simeone²,

Durisi³

2020

Preprint

View full text Add to dashboard Cite

Meta-learning automatically infers an inductive bias by observing data from a number of related tasks. The inductive bias is encoded by hyperparameters that determine aspects of the model class or training algorithm, such as initialization or learning rate. Meta-learning assumes that the learning tasks belong to a task environment, and that tasks are drawn from the same task environment both during meta-training and meta-testing. This, however, may not hold true in practice. In this paper, we introduce the problem of transfer meta-learning, in which tasks are drawn from a target task environment during meta-testing that may differ from the source task environment observed during meta-training.Novel information-theoretic upper bounds are obtained on the transfer meta-generalization gap, which measures the difference between the meta-training loss, available at the meta-learner, and the average loss on meta-test data from a new, randomly selected, task in the target task environment. The first bound, on the average transfer meta-generalization gap, captures the meta-environment shift between source and target task environments via the KL divergence between source and target data distributions.The second, PAC-Bayesian bound, and the third, single-draw bound, account for this shift via the loglikelihood ratio between source and target task distributions. Furthermore, two transfer meta-learning solutions are introduced. For the first, termed Empirical Meta-Risk Minimization (EMRM), we derive bounds on the average optimality gap. The second, referred to as Information Meta-Risk Minimization (IMRM), is obtained by minimizing the PAC-Bayesian bound. IMRM is shown via experiments to potentially outperform EMRM.

show abstract

Meta-strategy for Learning Tuning Parameters with Guarantees

Meunier,

Alquier

2021

Preprint

View full text Add to dashboard Cite

Online gradient methods, like the online gradient algorithm (OGA), often depend on tuning parameters that are difficult to set in practice. We consider an online meta-learning scenario, and we propose a meta-strategy to learn these parameters from past tasks. Our strategy is based on the minimization of a regret bound. It allows to learn the initialization and the step size in OGA with guarantees. We provide a regret analysis of the strategy in the case of convex losses. It suggests that, when there are parameters θ1, . . . , θT solving well tasks 1, . . . , T respectively and that are close enough one to each other, our strategy indeed improves on learning each task in isolation.

show abstract

PACOH: Bayes-Optimal Meta-Learning with PAC-Guarantees

Cited by 8 publications

References 0 publications

Transfer Bayesian Meta-Learning Via Weighted Free Energy Minimization

Transfer Bayesian Meta-Learning Via Weighted Free Energy Minimization

Transfer Meta-Learning: Information-Theoretic Bounds and Information Meta-Risk Minimization

Meta-strategy for Learning Tuning Parameters with Guarantees

Contact Info

Product

Resources

About