Triple-GAIL: A Multi-Modal Imitation Learning Framework with Generative Adversarial Nets

Abstract

Generative adversarial imitation learning (GAIL) has shown promising results by taking advantage of generative adversarial nets, especially in the field of robot learning. However, the requirement of isolated single modal demonstrations limits the scalability of the approach to real world scenarios such as autonomous vehicles' demand for a proper understanding of human drivers' behavior. In this paper, we propose a novel multi-modal GAIL framework, named Triple-GAIL, that is able to learn skill selection and imitation jointly from both expert demonstrations and continuously generated experiences with data augmentation purpose by introducing an auxiliary skill selector. We provide theoretical guarantees on the convergence to optima for both of the generator and the selector respectively. Experiments on real driver trajectories and realtime strategy game datasets demonstrate that Triple-GAIL can better fit multi-modal behaviors close to the demonstrators and outperforms state-of-the-art methods.

A New Approach to Training Multiple Cooperative Agents for Autonomous Driving

“…The numbers of hidden units of the two fully connected layers are 300 and 600, respectively, and tanh is used as the activation function. The goal of the joint discriminator network is to maximize the function shown in (5).…”

“…Reinforcement learning methods, which can optimize sequential decisions, have shown excellent performance on the driving decisionmaking of a single car agent [1] [2]. Imitation learning methods, which directly learn the behavior of a driver from recorded driving experiences and generate the driving policies, provide alternative solutions to autonomous driving [3]- [5]. Therefore, multi-agent systems are expected to be constructed to achieve autonomous driving through multi-agent reinforcement learning [6] or imitation learning [7].…”

HKGAIL: Policy shaping via integrating human knowledge with generative adversarial imitation learning

“…Li et al [20] proposed the InfoGAIL framework to infer potential modal variables by maximizing the mutual information between potential variables and state-action pairs. Fei et al [8] proposed Triple-GAIL, which allows co-learning from expert demonstrations adaptive skill selection and imitation, and generates experience by introducing auxiliary skill selectors. Bhattacharyya [21] proposed PS-GAIL, which extends Gail through a parameter sharing approach based on course learning.…”

“…In contrast, IRL focuses on finding the reward function that best explains the demonstrated behaviour, however, the function search is ill-adapted because the demonstrated behaviour may correspond to multiple reward functions, and IRL methods inevitably suffer from intensive computation, which limits their efficiency in high-dimensional scenarios. Generative adversarial imitation learning (GAIL) [7] has become a very popular model-free imitation learning framework that can extract policies directly from expert demonstrations, overcoming cascading errors caused by behavioural cloning (BC) and reducing the computational burden of inverse reinforcement learning (IRL) [8]. Work has been done to show that GAIL can perform well in complex large-scale problems such as autonomous driving [4], simulation, and real robot manipulation [9].…”

“…• Similar to Triple-Gail [8], a new adversarial game framework that extends the original GAIL with the addition of a human decision maker is proposed. The human decision maker and generator in HKGAIL generate policy distributions given expert state-action pairs, while the discriminator distinguishes whether a policy comes from an expert presentation or not.…”