Importance Weighted Transfer of Samples in Reinforcement Learning

Tirinzoni, Andrea; Sessa, Andrea; Pirotta, Matteo; Restelli, Marcello

doi:10.48550/arxiv.1805.10886

Cited by 2 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The basic idea of instance transfer algorithms is that the transfer of teacher samples may improve the learning on student tasks. Lazaric et al (2008) and Tirinzoni et al (2018) selectively transfer samples on the basis of the compliance between tasks in a model-free algorithm, while Taylor et al (2008a) studies how a model-based algorithm can benefit from samples coming from the teacher task.…”

Section: Related Work: Transfer Reinforcement Learningmentioning

confidence: 99%

REPAINT: Knowledge Transfer in Deep Reinforcement Learning

Tao,

Genc,

Chung

et al. 2020

Preprint

View full text Add to dashboard Cite

Accelerating the learning processes for complex tasks by leveraging previously learned tasks has been one of the most challenging problems in reinforcement learning, especially when the similarity between source and target tasks is low or unknown. In this work, we propose a REPresentation-And-INstance Transfer algorithm (REPAINT) for deep actor-critic reinforcement learning paradigm. In representation transfer, we adopt a kickstarted training method using a pre-trained teacher policy by introducing an auxiliary cross-entropy loss. In instance transfer, we develop a sampling approach, i.e., advantage-based experience replay, on transitions collected following the teacher policy, where only the samples with high advantage estimates are retained for policy update. We consider both learning an unseen target task by transferring from previously learned teacher tasks and learning a partially unseen task composed of multiple sub-tasks by transferring from a pre-learned teacher sub-task. In several benchmark experiments, REPAINT significantly reduces the total training time and improves the asymptotic performance compared to training with no prior knowledge and other baselines.

show abstract

Section: Related Work: Transfer Reinforcement Learningmentioning

confidence: 99%

REPAINT: Knowledge Transfer in Deep Reinforcement Learning

Tao,

Genc,

Chung

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Depending on what kind of knowledge representation is being transferred, we have different TL algorithms in the related literature. Therefore, in order to perform the transfer, we may have algorithms leveraging policies or options [9,17], samples [33,20,38,36], features [2,21], value-functions [35,37] or parameters [15,1,26,8].…”

Section: Introductionmentioning

confidence: 99%

Time-Variant Variational Transfer for Value Functions

Canonaco,

Soprani,

Roveri

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

In most of the transfer learning approaches to reinforcement learning (RL) the distribution over the tasks is assumed to be stationary. Therefore, the target and source tasks are i.i.d. samples of the same distribution. In the context of this work, we consider the problem of transferring value functions through a variational method when the distribution that generates the tasks is time-variant, proposing a solution that leverages this temporal structure inherent in the task generating process. Furthermore, by means of a finite-sample analysis, the previously mentioned solution is theoretically compared to its time-invariant version. Finally, we will provide an experimental evaluation of the proposed technique with three distinct temporal dynamics in three different RL environments. * equal contribution Preprint. Under review.

show abstract

Importance Weighted Transfer of Samples in Reinforcement Learning

Cited by 2 publications

References 0 publications

REPAINT: Knowledge Transfer in Deep Reinforcement Learning

REPAINT: Knowledge Transfer in Deep Reinforcement Learning

Time-Variant Variational Transfer for Value Functions

Contact Info

Product

Resources

About