PLAS: Latent Action Space for Offline Reinforcement Learning

Zhou, Wenxuan; Bajracharya, Sujay; Held, David

doi:10.48550/arxiv.2011.07213

Cited by 11 publications

(10 citation statements)

References 13 publications

(26 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The major challenge in offline RL is distribution shift [20,37,39], where the learned policy might generate out-of-distribution actions, resulting in erroneous value backups. Prior offline RL methods address this issue by regularizing the learned policy to be "close" to the behavior policy [20,50,29,82,93,37,68,57], through variants of importance sampling [59,74,49,75,54], via uncertainty quantification on Q-values [2,37,82,43], by learning conservative Q-functions [39,36], and with model-based training with a penalty on out-ofdistribution states [34,91,53,4,76,60,42,92]. While current benchmarks in offline RL [19,25] contain datasets that involve multi-task structure, existing offline RL methods do not leverage the shared structure of multiple tasks and instead train each individual task from scratch.…”

Section: Related Workmentioning

confidence: 99%

Conservative Data Sharing for Multi-Task Offline Reinforcement Learning

Yu¹,

Kumar²,

Chebotar³

et al. 2021

Preprint

View full text Add to dashboard Cite

Offline reinforcement learning (RL) algorithms have shown promising results in domains where abundant pre-collected data is available. However, prior methods focus on solving individual problems from scratch with an offline dataset without considering how an offline RL agent can acquire multiple skills. We argue that a natural use case of offline RL is in settings where we can pool large amounts of data collected in various scenarios for solving different tasks, and utilize all of this data to learn behaviors for all the tasks more effectively rather than training each one in isolation. However, sharing data across all tasks in multi-task offline RL performs surprisingly poorly in practice. Thorough empirical analysis, we find that sharing data can actually exacerbate the distributional shift between the learned policy and the dataset, which in turn can lead to divergence of the learned policy and poor performance. To address this challenge, we develop a simple technique for data-sharing in multi-task offline RL that routes data based on the improvement over the task-specific data. We call this approach conservative data sharing (CDS), and it can be applied with multiple single-task offline RL methods. On a range of challenging multi-task locomotion, navigation, and vision-based robotic manipulation problems, CDS achieves the best or comparable performance compared to prior offline multi-task RL methods and previous data sharing approaches.Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

Conservative Data Sharing for Multi-Task Offline Reinforcement Learning

Yu¹,

Kumar²,

Chebotar³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The main challenge of offline RL is the distributional shift between the learned policy and the behavior policy (Fujimoto et al, 2018), which can cause erroneous value backups. To address this issue, prior methods have constrained the learned policy to be close to the behavior policy via policy regularization (Liu et al, 2020;Wu et al, 2019;Kumar et al, 2019;Zhou et al, 2020;Ghasemipour et al, 2021;Fujimoto & Gu, 2021), conservative value functions (Kumar et al, 2020), and model-based training with conservative penalties (Yu et al, 2020c;Kidambi et al, 2020;Swazinna et al, 2020;Lee et al, 2021;Yu et al, 2021b). Unlike these prior works, we study how unlabeled data can be incorporated into the offline RL framework.…”

Section: Related Workmentioning

confidence: 99%

How to Leverage Unlabeled Data in Offline Reinforcement Learning

Yu¹,

Kumar²,

Chebotar³

et al. 2022

Preprint

View full text Add to dashboard Cite

Offline reinforcement learning (RL) can learn control policies from static datasets but, like standard RL methods, it requires reward annotations for every transition. In many cases, labeling large datasets with rewards may be costly, especially if those rewards must be provided by human labelers, while collecting diverse unlabeled data might be comparatively inexpensive. How can we best leverage such unlabeled data in offline RL? One natural solution is to learn a reward function from the labeled data and use it to label the unlabeled data. In this paper, we find that, perhaps surprisingly, a much simpler method that simply applies zero rewards to unlabeled data leads to effective data sharing both in theory and in practice, without learning any reward model at all. While this approach might seem strange (and incorrect) at first, we provide extensive theoretical and empirical analysis that illustrates how it trades off reward bias, sample complexity and distributional shift, often leading to good results. We characterize conditions under which this simple strategy is effective, and further show that extending it with a simple reweighting approach can further alleviate the bias introduced by using incorrect reward labels. Our experiments confirm these findings in simulated robotic locomotion, navigation, and manipulation settings.

show abstract

“…The use of generative models in robot learning has become popular in recent years [2,3,[11][12][13][42][43][44][45][46][47][48][49][50][51][52] because of their low-dimensional and regularized latent spaces. However, latent variable generative models are mainly studied to train a long-term state prediction model that is used in the context of trajectory optimization and model-based reinforcement learning [47][48][49][50][51][52].…”

Section: Related Workmentioning

confidence: 99%

Training and Evaluation of Deep Policies using Reinforcement Learning and Generative Models

Ghadirzadeh¹,

Poklukar²,

Arndt³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present a data-efficient framework for solving sequential decision-making problems which exploits the combination of reinforcement learning (RL) and latent variable generative models. The framework, called GenRL, trains deep policies by introducing an action latent variable such that the feed-forward policy search can be divided into two parts: (i) training a sub-policy that outputs a distribution over the action latent variable given a state of the system, and (ii) unsupervised training of a generative model that outputs a sequence of motor actions conditioned on the latent action variable. GenRL enables safe exploration and alleviates the data-inefficiency problem as it exploits prior knowledge about valid sequences of motor actions. Moreover, we provide a set of measures for evaluation of generative models such that we are able to predict the performance of the RL policy training prior to the actual training on a physical robot. We experimentally determine the characteristics of generative models that have most influence on the performance of the final policy training on two robotics tasks: shooting a hockey puck and throwing a basketball. Furthermore, we empirically demonstrate that GenRL is the only method which can safely and efficiently solve the robotics tasks compared to two state-of-the-art RL methods.

show abstract

PLAS: Latent Action Space for Offline Reinforcement Learning

Cited by 11 publications

References 13 publications

Conservative Data Sharing for Multi-Task Offline Reinforcement Learning

Conservative Data Sharing for Multi-Task Offline Reinforcement Learning

How to Leverage Unlabeled Data in Offline Reinforcement Learning

Training and Evaluation of Deep Policies using Reinforcement Learning and Generative Models

Contact Info

Product

Resources

About