Incremental Learning-to-Learn with Statistical Guarantees

Denevi, Giulia; Ciliberto, Carlo; Stamos, Dimitris; Pontil, Massimiliano

doi:10.48550/arxiv.1803.08089

Cited by 13 publications

(14 citation statements)

References 14 publications

(49 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[14,19,24,26,10]. We also note that there are analyses for other representation learning schemes [5,31,20,2,15], which are beyond the scope of this paper.…”

Section: Related Workmentioning

confidence: 99%

On the Power of Multitask Representation Learning in Linear MDP

Lu,

Huang,

2021

Preprint

View full text Add to dashboard Cite

While multitask representation learning has become a popular approach in reinforcement learning (RL), theoretical understanding of why and when it works remains limited. This paper presents analyses for the statistical benefit of multitask representation learning in linear Markov Decision Process (MDP) under a generative model. In this paper, we consider an agent to learn a representation function φ out of a function class Φ from T source tasks with N data per task, and then use the learned φ to reduce the required number of sample for a new task. We first discover a Least-Activated-Feature-Abundance (LAFA) criterion, denoted as κ, with which we prove that a straightforward least-square algorithm learns a policyHere H is the planning horizon,and n is the number of samples for the new task. In multitask RL, N T is usually sufficiently large and the bound is dominated by the second term. Thus the required n is O(κdH 4 ) for the sub-optimality to be close to zero, which is much smaller than O(C(Φ) 2 κdH 4 ) in the setting without multitask representation learning, whose sub-optimality gap is Õ H 2 κC(Φ) 2 d n . This theoretically explains the power of multitask representation learning in reducing sample complexity. Further, we note that to ensure high sample efficiency, the LAFA criterion κ should be small. In fact, κ varies widely in magnitude depending on the different sampling distribution for new task. This indicates adaptive sampling technique is important to make κ solely depend on d. Finally, we provide empirical results of a noisy grid-world environment to corroborate our theoretical findings.Preprint. Under review.

show abstract

“…[14,19,24,26,10]. We also note that there are analyses for other representation learning schemes [5,31,20,2,15], which are beyond the scope of this paper.…”

Section: Related Workmentioning

confidence: 99%

On the Power of Multitask Representation Learning in Linear MDP

Lu,

Huang,

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Inspired by MAML, a line of gradient-based meta-learning algorithms have been widely used in practice Nichol et al (2018); Al-Shedivat et al (2017); Jerfel et al (2018). Much follow-up work focused on online-setting with regret bounds (Denevi et al, 2018;Finn et al, 2019;Khodak et al, 2019;Balcan et al, 2015;Alquier et al, 2017;Bullins et al, 2019;Pentina & Lampert, 2014).…”

Section: Related Workmentioning

confidence: 99%

“…We show that fine-tuning quickly adapts to new tasks, requiring fewer samples in certain cases compared to methods using "frozen representation" objectives (as studied in Du et al (2020), and will be formalized in Section 2.2). To the best of our knowledge, no prior studies exist beyond fine-tuning a linear model (Denevi et al, 2018;Konobeev et al, 2020;Collins et al, 2020a;Lee et al, 2020) or only the task-specific layers (Du et al, 2020;Tripuraneni et al, 2020b,a;Mu et al, 2020). In particular, our work can be viewed as a continuation of the work presented in Tripuraneni et al (2020b), where the authors have acknowledged that the framework does not incorporate representation fine-tuning, and thus is a promising line of future work.…”

Section: Introductionmentioning

confidence: 99%

How Fine-Tuning Allows for Effective Meta-Learning

Chua¹,

Liu²,

Lee³

2021

Preprint

View full text Add to dashboard Cite

Representation learning has been widely studied in the context of meta-learning, enabling rapid learning of new tasks through shared representations. Recent works such as MAML have explored using fine-tuning-based metrics, which measure the ease by which fine-tuning can achieve good performance, as proxies for obtaining representations. We present a theoretical framework for analyzing representations derived from a MAML-like algorithm, assuming the available tasks use approximately the same underlying representation. We then provide risk bounds on the best predictor found by fine-tuning via gradient descent, demonstrating that the algorithm can provably leverage the shared structure. The upper bound applies to general function classes, which we demonstrate by instantiating the guarantees of our framework in the logistic regression and neural network settings. In contrast, we establish the existence of settings where any algorithm, using a representation trained with no consideration for task-specific fine-tuning, performs as well as a learner with no access to source tasks in the worst case. This separation result underscores the benefit of fine-tuningbased methods, such as MAML, over methods with "frozen representation" objectives in few-shot learning.

show abstract

“…This thesis focuses on the first MAML algorithms, but the techniques here can be extended to analyze the Hessian-free multi-step MAML. Alternatively to meta-initialization algorithms such as MAML, meta-regularization approaches aim to learn a good bias for a regularized empirical risk minimization problem for intra-task learning [2,22,21,20,104,8,132]. [8] formalized a connection between meta-initialization and meta-regularization from an online learning perspective.…”

Section: Related Workmentioning

confidence: 99%

Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis

2021

Preprint

View full text Add to dashboard Cite

Bilevel optimization has become a powerful framework in a variety of machine learning applications including signal processing, meta-learning, hyperparameter optimization, reinforcement learning and network architecture search. There are generally two classes of bilevel optimization formulations for modern machine learning: 1) problem-based bilevel optimization, whose inner-level problem is formulated as finding a minimizer of a given loss function; and 2) algorithm-based bilevel optimization, whose inner-level solution is an output of a fixed algorithm. For the first problem class, two popular types of gradient-based algorithms have been proposed to estimate the gradient of the outer-level objective (hypergradient) via approximate implicit differentiation (AID) and iterative differentiation (ITD). Algorithms for the second problem class include the popular model-agnostic meta-learning (MAML) and almost no inner loop (ANIL). Although bilevel optimization algorithms have been widely used, their convergence rate and fundamental limitations have not been well explored.

show abstract

Incremental Learning-to-Learn with Statistical Guarantees

Cited by 13 publications

References 14 publications

On the Power of Multitask Representation Learning in Linear MDP

On the Power of Multitask Representation Learning in Linear MDP

How Fine-Tuning Allows for Effective Meta-Learning

Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis

Contact Info

Product

Resources

About