Generalization and Computation for Policy Classes of Generative Adversarial Imitation Learning

Zhou, Yubin; Zhang, Yangchun; Liu, Xiaowei; Wang, Wanying; Che, Zhengping; Xu, Zhiyuan; Tang, Jian; Peng, Yaxin

doi:10.1007/978-3-031-14714-2_27

Cited by 2 publications

(7 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Within IL, IRL [2] has emerged as a significant approach. IRL, which focuses on inferring a reward function that leads to an optimal policy aligned with expert behavior, has shown remarkable effectiveness and attracted much attention in recent years [11,20,21,14,15,22]. The two primary challenges in IRL are the imitation of the policy and the recovery of reward functions [11,23].…”

Section: Related Workmentioning

confidence: 99%

“…IRL, by implementing distribution matching for imitation, has made substantial progress [4,20,21,15,22]. Drawing on the principles of generative adversarial networks (GANs) [24] used in IL, generative adversarial imitation learning (GAIL) [4] integrates a discriminator-generated reward system and a policy gradient approach to effectively mimic an expert RL policy.…”

Section: Policy Imitationmentioning

confidence: 99%

“…ValueDICE [21] sidesteps the methods of explicitly learning or defining rewards and directly acquires a Q-function within the distribution ratio objective. Subsequently, Zhou et al [15] conducted an in-depth analysis of various issues ranging from generalization to computation in GAIL and proposed the greedy distributional soft gradient (GDSG) algorithm by the distributional soft actor-critic (DSAC) [25,26] update for the policy. Recently, coupled flow imitation learning (CFIL) [22] effectively integrates the log ratio estimator, techniques for smoothing and regularization, the employment of flows and the use of behavioral cloning (BC) [1,27] graphs, thereby yielding strong performance with a single expert trajectory.…”

Section: Policy Imitationmentioning

confidence: 99%

“…To enhance sample efficiency against AIRL, we substitute its built-in algorithm with soft actor-critic (SAC) [12] during the policy optimization process, referred to as SAC-AIRL. This choice is influenced by SAC's superior sample efficiency over the original trust region policy optimization (TRPO) algorithm [13] in IL [14,15]. As anticipated, SAC-AIRL demonstrates a significant improvement in imitation performance (Details can be found in Section 4).…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Improve generated adversarial imitation learning with reward variance regularization

Zhang

Fan-Ming

2022

Mach Learn

View full text Add to dashboard Cite

Imitation learning aims at recovering expert policies from limited demonstration data. Generative Adversarial Imitation Learning (GAIL) employs the generative adversarial learning framework for imitation learning and has shown great potentials. GAIL and its variants, however, are found highly sensitive to hyperparameters and hard to converge well in practice. One key issue is that the supervised learning discriminator has a much faster learning speed than the reinforcement learning generator, making the generator gradient vanishing. Although GAIL is formulated as a zero-sum adversarial game, the ultimate goal of GAIL is to learn the generator, thus the discriminator should play the role more like a teacher rather than a real opponent. Therefore, the learning of the discriminator should consider how the generator could learn. In this paper, we disclose that enhancing the gradient of the generator training is equivalent to increase the variance of the fake reward provided by the discriminator output. We thus propose an improved version of GAIL, GAIL-VR, in which the discriminator also learns to avoid generator gradient vanishing through regularization of the fake rewards variance. Experiments in various tasks, including locomotion tasks and Atari games, indicate that GAIL-VR can improve the training stability and imitation scores.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Policy Imitationmentioning

confidence: 99%

Section: Policy Imitationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Improve generated adversarial imitation learning with reward variance regularization

Zhang

Fan-Ming

2022

Mach Learn

View full text Add to dashboard Cite

show abstract

“…The GAIL can be bifurcated into two genres: stochastic policy algorithms and deterministic policy algorithms, namely DE-GAIL (Kostrikov et al 2019;Zuo et al 2020) and ST-GAIL (Ho and Ermon 2016;Zhou et al 2022). The ST-GAIL with stochastic policy guarantees global convergence in high-dimensional environments, outperforming traditional Inverse Reinforcement Learning (IRL) methods (Ng, Russell et al 2000;Ziebart et al 2008;Boularias, Kober, and Peters 2011).…”

Section: Introductionmentioning

confidence: 99%

Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspective

Wang,

Zhu,

Zhou

et al. 2024

AAAI

View full text Add to dashboard Cite

Generative Adversarial Imitation Learning (GAIL) stands as a cornerstone approach in imitation learning. This paper investigates the gradient explosion in two types of GAIL: GAIL with deterministic policy (DE-GAIL) and GAIL with stochastic policy (ST-GAIL). We begin with the observation that the training can be highly unstable for DE-GAIL at the beginning of the training phase and end up divergence. Conversely, the ST-GAIL training trajectory remains consistent, reliably converging. To shed light on these disparities, we provide an explanation from a theoretical perspective. By establishing a probabilistic lower bound for GAIL, we demonstrate that gradient explosion is an inevitable outcome for DE-GAIL due to occasionally large expert-imitator policy disparity, whereas ST-GAIL does not have the issue with it. To substantiate our assertion, we illustrate how modifications in the reward function can mitigate the gradient explosion challenge. Finally, we propose CREDO, a simple yet effective strategy that clips the reward function during the training phase, allowing the GAIL to enjoy high data efficiency and stable trainability.

show abstract

Generalization and Computation for Policy Classes of Generative Adversarial Imitation Learning

Cited by 2 publications

References 11 publications

Improve generated adversarial imitation learning with reward variance regularization

Improve generated adversarial imitation learning with reward variance regularization

Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspective

Contact Info

Product

Resources

About