LASER: Learning a Latent Action Space for Efficient Reinforcement Learning

Allshire, Arthur; Martí­n-Martí­n, Roberto; Lin, Charles C.; Manuel, Shawn; Savarese, Silvio; Garg, Animesh

doi:10.1109/icra48506.2021.9561232

Cited by 22 publications

(15 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Related Workmentioning

confidence: 99%

“…Learning latent action representations is studied in RL [26] and offline RL [27,28]. Allshire et al [26] proposed a conditional variational encoder-decoder (CVAE) model to learn latent action space where generated latent actions are decoded before applying on the original robot state. van der Pol et al [29] learns latent spaces with equivariant maps to show that optimal policies are equivalent but the application is limited to small discrete systems where value iteration can be applied.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Latent Policies for Adversarial Imitation Learning

Wang¹,

Karnwal²,

Atanasov³

2022

Preprint

View full text Add to dashboard Cite

This paper considers learning robot locomotion and manipulation tasks from expert demonstrations. Generative adversarial imitation learning (GAIL) trains a discriminator that distinguishes expert from agent transitions, and in turn use a reward defined by the discriminator output to optimize a policy generator for the agent. This generative adversarial training approach is very powerful but depends on a delicate balance between the discriminator and the generator training. In high-dimensional problems, the discriminator training may easily overfit or exploit associations with task-irrelevant features for transition classification. A key insight of this work is that performing imitation learning in a suitable latent task space makes the training process stable, even in challenging high-dimensional problems. We use an action encoder-decoder model to obtain a low-dimensional latent action space and train a LAtent Policy using Adversarial imitation Learning (LAPAL). The encoder-decoder model can be trained offline from state-action pairs to obtain a task-agnostic latent action representation or online, simultaneously with the discriminator and generator training, to obtain a task-aware latent action representation. We demonstrate that LAPAL training is stable, with nearmonotonic performance improvement, and achieves expert performance in most locomotion and manipulation tasks, while a GAIL baseline converges slower and does not achieve expert performance in high-dimensional environments.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Latent Policies for Adversarial Imitation Learning

Wang¹,

Karnwal²,

Atanasov³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Despite these efforts, today's DRL methods still struggle in longhorizon robotic tasks due to the exploration burden of learning from scratch. A growing amount of work has examined the use of offline data to alleviate the exploration burden in DRL, namely through demonstration-guided RL [17,45,55], learned behavioral priors [54,61] and action spaces [1,2] from demonstrations, and offline RL [14,15,32,39]. While promising, these methods can be difficult to scale up due to the costs of acquiring offline data.…”

Section: Related Workmentioning

confidence: 99%

“…x [2]] and we encourage the agent to specify the reaching parameters x reach to be within a threshold of a set of keypoints P :…”

Section: Facilitating Exploration With Affordancesmentioning

confidence: 99%

“…Such objects comprise objects that the robot was already holding before step (1) and that the robot subsequently lifted into the air during step (1). For such objects, we compute the pose of the object as the final robot pose in addition to the relative pose difference of the robot end effector and object during step (2). In addition to handling occlusions, we noticed that the pose estimation system routinely made small errors when estimating the position and orientation of the objects.…”

Section: J Task Sketch Transfer Experimentsmentioning

confidence: 99%

See 1 more Smart Citation

Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks

Nasiriany¹,

Huihan²,

Zhu³

2021

Preprint

View full text Add to dashboard Cite

Realistic manipulation tasks require a robot to interact with an environment with a prolonged sequence of motor actions. While deep reinforcement learning methods have recently emerged as a promising paradigm for automating manipulation behaviors, they usually fall short in long-horizon tasks due to the exploration burden. This work introduces MAnipulation Primitive-augmented reinforcement LEarning (MAPLE), a learning framework that augments standard reinforcement learning algorithms with a pre-defined library of behavior primitives. These behavior primitives are robust functional modules specialized in achieving manipulation goals, such as grasping and pushing. To use these heterogeneous primitives, we develop a hierarchical policy that involves the primitives and instantiates their executions with input parameters. We demonstrate that MAPLE outperforms baseline approaches by a significant margin on a suite of simulated manipulation tasks. We also quantify the compositional structure of the learned behaviors and highlight our method's ability to transfer policies to new task variants and to physical hardware. Videos and code are available at https://ut-austin-rpl.github. io/maple

show abstract

GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Model

Xie

Babich

et al. 2022

Springer Proceedings in Advanced Robotics

View full text Add to dashboard Cite

LASER: Learning a Latent Action Space for Efficient Reinforcement Learning

Cited by 22 publications

References 18 publications

Latent Policies for Adversarial Imitation Learning

Latent Policies for Adversarial Imitation Learning

Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks

GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Model

Contact Info

Product

Resources

About