Stateless neural meta-learning using second-order gradients

Huisman, Mike; Plaat, Aske; Rijn, Jan N. van

doi:10.1007/s10994-022-06210-y

Cited by 5 publications

(2 citation statements)

References 19 publications

(22 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These methods aim to metalearn good settings for various hyperparameters, such as the initialization parameters, such that new tasks can be learned quickly using optimization methods. These methods vary from regular stochastic gradient descent, as used in MAML (Finn et al, 2017) and Reptile (Nichol et al, 2018), to meta-learned procedures where a network updates the weights of a base-learner (Ravi et al, 2017;Andrychowicz et al, 2016;Li et al, 2017;Rusu et al, 2019;Li & Malik, 2018;Huisman et al, 2022). SAP aims to learn good initialization parameters such that new tasks can be learned quickly with regular gradient descent, similar to MAML.…”

Section: Optimization-based Meta-learningmentioning

confidence: 99%

“…Gradient-based meta-learning methods struggle to scale well to deep networks as recent work suggests that simple pre-training and fine-tuning of the output layer (Tian et al, 2020;Chen et al, 2021;Huisman et al, 2021a) can yield superior performance on common fewshot image classification benchmarks. This is also the reason, besides searching for energyefficient few-shot learners, that in our experiments we focus on relatively shallow backbones that adapt all layers when learning new tasks, instead of only the output layer.…”

Section: Future Workmentioning

confidence: 99%

See 1 more Smart Citation

Subspace Adaptation Prior for Few-Shot Learning

Huisman,

Plaat,

van Rijn

2023

Mach Learn

Self Cite

View full text Add to dashboard Cite

Gradient-based meta-learning techniques aim to distill useful prior knowledge from a set of training tasks such that new tasks can be learned more efficiently with gradient descent. While these methods have achieved successes in various scenarios, they commonly adapt all parameters of trainable layers when learning new tasks. This neglects potentially more efficient learning strategies for a given task distribution and may be susceptible to overfitting, especially in few-shot learning where tasks must be learned from a limited number of examples. To address these issues, we propose Subspace Adaptation Prior (SAP), a novel gradient-based meta-learning algorithm that jointly learns good initialization parameters (prior knowledge) and layer-wise parameter subspaces in the form of operation subsets that should be adaptable. In this way, SAP can learn which operation subsets to adjust with gradient descent based on the underlying task distribution, simultaneously decreasing the risk of overfitting when learning new tasks. We demonstrate that this ability is helpful as SAP yields superior or competitive performance in few-shot image classification settings (gains between 0.1% and 3.9% in accuracy). Analysis of the learned subspaces demonstrates that low-dimensional operations often yield high activation strengths, indicating that they may be important for achieving good few-shot learning performance. For reproducibility purposes, we publish all our research code publicly.

show abstract

Section: Optimization-based Meta-learningmentioning

confidence: 99%

Section: Future Workmentioning

confidence: 99%

Subspace Adaptation Prior for Few-Shot Learning

Huisman,

Plaat,

van Rijn

2023

Mach Learn

Self Cite

View full text Add to dashboard Cite

show abstract

Triplet MAML for Few-Shot Classification Problems

Gülcü,

Özkan,

Kuş

et al. 2023

Communications in Computer and Information Science

View full text Add to dashboard Cite

Understanding transfer learning and gradient-based meta-learning techniques

Huisman,

Plaat,

van Rijn

2023

Mach Learn

View full text Add to dashboard Cite

Deep neural networks can yield good performance on various tasks but often require large amounts of data to train them. Meta-learning received considerable attention as one approach to improve the generalization of these networks from a limited amount of data. Whilst meta-learning techniques have been observed to be successful at this in various scenarios, recent results suggest that when evaluated on tasks from a different data distribution than the one used for training, a baseline that simply finetunes a pre-trained network may be more effective than more complicated meta-learning techniques such as MAML, which is one of the most popular meta-learning techniques. This is surprising as the learning behaviour of MAML mimics that of finetuning: both rely on re-using learned features. We investigate the observed performance differences between finetuning, MAML, and another meta-learning technique called Reptile, and show that MAML and Reptile specialize for fast adaptation in low-data regimes of similar data distribution as the one used for training. Our findings show that both the output layer and the noisy training conditions induced by data scarcity play important roles in facilitating this specialization for MAML. Lastly, we show that the pre-trained features as obtained by the finetuning baseline are more diverse and discriminative than those learned by MAML and Reptile. Due to this lack of diversity and distribution specialization, MAML and Reptile may fail to generalize to out-of-distribution tasks whereas finetuning can fall back on the diversity of the learned features.

show abstract

Stateless neural meta-learning using second-order gradients

Cited by 5 publications

References 19 publications

Subspace Adaptation Prior for Few-Shot Learning

Subspace Adaptation Prior for Few-Shot Learning

Triplet MAML for Few-Shot Classification Problems

Understanding transfer learning and gradient-based meta-learning techniques

Contact Info

Product

Resources

About