Teaching with Commentaries

Raghu, Aniruddh; Raghu, Maithra; Kornblith, Simon; Duvenaud, David; Hinton, Geoffrey E.

doi:10.48550/arxiv.2011.03037

Cited by 3 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…FT Hypergradient: For the gradient through FT, we acknowledge that differentiating through only one step could, in theory, produce biased hypergradients. However, several prior works on meta-learning various structures similar to what we consider [51,54,42,49,29] did not observe significant bias. Therefore, from an empirical standpoint, this bias is not necessarily expected to be a significant issue.…”

Section: A Notation and Acronymsmentioning

confidence: 66%

“…Applications of Nested Optimization: Many prior works frame learning as nested optimization, including few-shot learning [16,1,17,55,21,58,53,75,31,38], neural network teaching [14,15,62,54], learning data augmentation and reweighting strategies [32,22,57,60,29], and auxiliary task learning [49,51,39]. The majority of this work studies nested optimization in the standard one-stage supervised learning paradigm, unlike our setting: the two-stage PT & FT problem.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Meta-Learning to Improve Pre-Training

Raghu¹,

Lorraine²,

Kornblith³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Pre-training (PT) followed by fine-tuning (FT) is an effective method for training neural networks, and has led to significant performance improvements in many domains. PT can incorporate various design choices such as task and data reweighting strategies, augmentation policies, and noise models, all of which can significantly impact the quality of representations learned. The hyperparameters introduced by these strategies therefore must be tuned appropriately. However, setting the values of these hyperparameters is challenging. Most existing methods either struggle to scale to high dimensions, are too slow and memory-intensive, or cannot be directly applied to the two-stage PT and FT learning process. In this work, we propose an efficient, gradient-based algorithm to meta-learn PT hyperparameters. We formalize the PT hyperparameter optimization problem and propose a novel method to obtain PT hyperparameter gradients by combining implicit differentiation and backpropagation through unrolled optimization. We demonstrate that our method improves predictive performance on two real-world domains. First, we optimize high-dimensional task weighting hyperparameters for multitask pre-training on protein-protein interaction graphs and improve AUROC by up to 3.9%. Second, we optimize a data augmentation neural network for self-supervised PT with SimCLR on electrocardiography data and improve AUROC by up to 1.9%.The PT & FT paradigm introduces high-dimensional, complex PT hyperparameters, such as parameterized data augmentation policies used in contrastive representation learning [8,22] or the use of task, class, or instance weighting variables in multi-task PT to avoid negative transfer [70]. These hyperparameters can significantly affect the quality of pre-trained models [8], and thus finding techniques to set their values optimally is an important area of research.Choosing optimal PT hyperparameter values is challenging, and existing methods do not work well. Simple approaches such as random or grid search are inefficient since evaluating a hyperparameter setting requires performing the full, two-stage PT & FT optimization, which may be prohibitively computationally expensive. Gradient-free approaches, such as Bayesian optimization or evolutionary algorithms [33,61,47], are also limited in how well they scale to this setting. Gradient-based 35th Conference on Neural Information Processing Systems (NeurIPS 2021).

show abstract

Section: A Notation and Acronymsmentioning

confidence: 66%

Section: Related Workmentioning

confidence: 99%

Meta-Learning to Improve Pre-Training

Raghu¹,

Lorraine²,

Kornblith³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Auxiliary losses are pooled over all examples and training epochs and their effect is only known at validation/test time. We would need to use implicit gradients [29,35] to know their eventual effect on the final weights at the end of training. With tailoring, we can directly measure the effect of the meta-learned update on the same sample.…”

Section: Noether Networkmentioning

confidence: 99%

Noether Networks: Meta-Learning Useful Conserved Quantities

Alet¹,

Doblar²,

Zhou³

et al. 2021

Preprint

View full text Add to dashboard Cite

Progress in machine learning (ML) stems from a combination of data availability, computational resources, and an appropriate encoding of inductive biases. Useful biases often exploit symmetries in the prediction problem, such as convolutional networks relying on translation equivariance. Automatically discovering these useful symmetries holds the potential to greatly improve the performance of ML systems, but still remains a challenge. In this work, we focus on sequential prediction problems and take inspiration from Noether's theorem to reduce the problem of finding inductive biases to meta-learning useful conserved quantities. We propose Noether Networks: a new type of architecture where a meta-learned conservation loss is optimized inside the prediction function. We show, theoretically and experimentally, that Noether Networks improve prediction quality, providing a general framework for discovering inductive biases in sequential problems.

show abstract

“…A growing number of applications require learning in games, which generalize single-objective optimization. Common examples are GANs (Goodfellow et al, 2014), actor-critic models (Pfau & Vinyals, 2016), curriculum learning (Baker et al, 2019;Balduzzi et al, 2019;Sukhbaatar et al, 2018), hyperparameter optimization (Lorraine & Duvenaud, 2018;Lorraine et al, 2020;MacKay et al, 2019;Raghu et al, 2020), adversarial examples (Bose et al, 2020;Yuan et al, 2019), learning models (Rajeswaran et al, 2020;Abachi et al, 2020;Bacon et al, 2019), domain adversarial adaptation (Acuna et al, 2021), neural architecture search (Grathwohl et al, 2018;Adam & Lorraine, 2019), and meta-learning (Ren et al, 2018;.…”

Section: Introductionmentioning

confidence: 99%

Complex Momentum for Optimization in Games

Lorraine,

Acuna,

Vicol

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We generalize gradient descent with momentum for learning in differentiable games to have complex-valued momentum. We give theoretical motivation for our method by proving convergence on bilinear zero-sum games for simultaneous and alternating updates. Our method gives real-valued parameter updates, making it a dropin replacement for standard optimizers. We empirically demonstrate that complex-valued momentum can improve convergence in adversarial games-like generative adversarial networks-by showing we can find better solutions with an almost identical computational cost. We also show a practical generalization to a complex-valued Adam variant, which we use to train BigGAN to better inception scores on CIFAR-10.

show abstract

Teaching with Commentaries

Cited by 3 publications

References 0 publications

Meta-Learning to Improve Pre-Training

Meta-Learning to Improve Pre-Training

Noether Networks: Meta-Learning Useful Conserved Quantities

Complex Momentum for Optimization in Games

Contact Info

Product

Resources

About