OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matching

Hoshino, Hana; Ota, Kei; Kanezaki, Asako; Yokota, Rio

doi:10.1109/icra46639.2022.9811660

Cited by 4 publications

(4 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…MB-ERIL is one instantiation of this approach. Recent approaches corrected the issues around reusing data previously collected during training the discriminator [17], [18].…”

Section: Literature Review a Model-free Imitation Learningmentioning

confidence: 99%

Model-Based Imitation Learning Using Entropy Regularization of Model and Policy

Uchibe

2022

Preprint

View full text Add to dashboard Cite

Approaches based on generative adversarial networks for imitation learning are promising because they are sample efficient in terms of expert demonstrations. However, training a generator requires many interactions with the actual environment because model-free reinforcement learning is adopted to update a policy. To improve the sample efficiency using model-based reinforcement learning, we propose modelbased Entropy-Regularized Imitation Learning (MB-ERIL) under the entropy-regularized Markov decision process to reduce the number of interactions with the actual environment. MB-ERIL uses two discriminators. A policy discriminator distinguishes the actions generated by a robot from expert ones, and a model discriminator distinguishes the counterfactual state transitions generated by the model from the actual ones. We derive the structured discriminators so that the learning of the policy and the model is efficient. Computer simulations and real robot experiments show that MB-ERIL achieves a competitive performance and significantly improves the sample efficiency compared to baseline methods.

show abstract

“…MB-ERIL is one instantiation of this approach. Recent approaches corrected the issues around reusing data previously collected during training the discriminator [17], [18].…”

Section: Literature Review a Model-free Imitation Learningmentioning

confidence: 99%

Model-Based Imitation Learning Using Entropy Regularization of Model and Policy

Uchibe

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…MB-ERIL is one instantiation of this approach. Recent approaches corrected the issues around reusing data previously collected while training the discriminator [17], [18].…”

Section: A Model-free Imitation Learningmentioning

confidence: 99%

Model-Based Imitation Learning Using Entropy Regularization of Model and Policy

Uchibe

2022

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

“…One of the angles in this realm is to recover a robust reward [11,28,29,30]. The AIRL algorithm [11] provides for simultaneous learning of the reward and value function, which is robust to changes in dynamics.…”

Section: Reward Recoverymentioning

confidence: 99%

“…Based on the idea of distribution matching and AIRL, Hoshino et al [29] formulated off-policy inverse reinforcement learning (OPIRL), which not only improves sample efficiency but is able to generalize to unseen environments. Further, receding-horizon inverse reinforcement learning (RHIRL) [23] shows its superiority in scalability and robustness for high-dimensional, noisy, continuous systems with black-box dynamic models.…”

Section: Reward Recoverymentioning

confidence: 99%

Improve generated adversarial imitation learning with reward variance regularization

Zhang

Fan-Ming

2022

Mach Learn

View full text Add to dashboard Cite

Imitation learning aims at recovering expert policies from limited demonstration data. Generative Adversarial Imitation Learning (GAIL) employs the generative adversarial learning framework for imitation learning and has shown great potentials. GAIL and its variants, however, are found highly sensitive to hyperparameters and hard to converge well in practice. One key issue is that the supervised learning discriminator has a much faster learning speed than the reinforcement learning generator, making the generator gradient vanishing. Although GAIL is formulated as a zero-sum adversarial game, the ultimate goal of GAIL is to learn the generator, thus the discriminator should play the role more like a teacher rather than a real opponent. Therefore, the learning of the discriminator should consider how the generator could learn. In this paper, we disclose that enhancing the gradient of the generator training is equivalent to increase the variance of the fake reward provided by the discriminator output. We thus propose an improved version of GAIL, GAIL-VR, in which the discriminator also learns to avoid generator gradient vanishing through regularization of the fake rewards variance. Experiments in various tasks, including locomotion tasks and Atari games, indicate that GAIL-VR can improve the training stability and imitation scores.

show abstract

OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matching

Cited by 4 publications

References 9 publications

Model-Based Imitation Learning Using Entropy Regularization of Model and Policy

Model-Based Imitation Learning Using Entropy Regularization of Model and Policy

Model-Based Imitation Learning Using Entropy Regularization of Model and Policy

Improve generated adversarial imitation learning with reward variance regularization

Contact Info

Product

Resources

About