I²HRL: Interactive Influence-based Hierarchical Reinforcement Learning

Wang, Rundong; Rong, Yu; An, Bo; Rabinovich, Zinovi

doi:10.24963/ijcai.2020/433

Cited by 11 publications

(17 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In (Zhang et al, 2020) the large subgoal space issue was addressed by restricting the highlevel action space from the whole subgoal space using an adjacency constraint. In (Wang et al, 2020) high-level policy decision making is conditioned on the received low-level representation as well as the state of the environment to improve stationarity. Another solutions is to add a slowness objective to effectively learn the subgoal representation so that the low-level reward function varies in a stationary way (Li et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

Hierarchical Reinforcement Learning with Adversarially Guided Subgoals

Huiling¹,

Pajarinen²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Hierarchical reinforcement learning (HRL) proposes to solve difficult tasks by performing decision-making and control at successively higher levels of temporal abstraction. However, off-policy HRL often suffers from the problem of non-stationary high-level policy since the lowlevel policy is constantly changing. In this paper, we propose a novel HRL approach for mitigating the non-stationarity by adversarially enforcing the high-level policy to generate subgoals compatible with the current instantiation of the lowlevel policy. In practice, the adversarial learning is implemented by training a simple discriminator network concurrently with the high-level policy which determines the compatibility level of subgoals. Experiments with state-of-the-art algorithms show that our approach improves both HRL learning efficiency and overall performance in various challenging continuous control tasks.

show abstract

Section: Related Workmentioning

confidence: 99%

Hierarchical Reinforcement Learning with Adversarially Guided Subgoals

Huiling¹,

Pajarinen²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…For both SPR and OPR, we use permutation-invariant transformations, i.e., mainly use MLP then Mean-Reduce operation, and detailed implementation is provided in Appendix D.2. A few works also involve representation or embedding learning for RL policy in Multiagent Learning (Grover et al, 2018), Hierarchical RL (Wang et al, 2020), Policy Adaptation and Transfer (Hausman et al, 2018;Arnekvist et al, 2019;Raileanu et al, 2020;Harb et al, 2020). We found almost all of them belongs to the scope of SPR.…”

Section: Policy Representation Learningmentioning

confidence: 99%

“…Recent years, a few works involve representation or embedding learning for RL policy (Hausman et al, 2018;Grover et al, 2018;Arnekvist et al, 2019;Raileanu et al, 2020;Wang et al, 2020;Harb et al, 2020). We provide a brief review and summary for above works below.…”

Section: D5 a Review Of Related Work On Policy Representation/embeddi...mentioning

confidence: 99%

“…Such opponent policy representations are used for prediction of interaction outcomes for ad-hoc teams and are taken in policy network for some learning agent to facilitate the learning when cooperating or competing with unknown opponents. More recently, in Hierarchical RL (Wang et al, 2020), a representation is learned to model the low-level policy through generative loss mentioned above. The low-level policy representation is taken in high-level policy to counter the non-stationarity issue of co-learning of hierarchical policies.…”

Section: D5 a Review Of Related Work On Policy Representation/embeddi...mentioning

confidence: 99%

See 1 more Smart Citation

What About Inputing Policy in Value Function: Policy Representation and Policy-extended Value Function Approximator

Tang¹,

Meng²,

Hao³

et al. 2020

Preprint

View full text Add to dashboard Cite

The value function lies in the heart of Reinforcement Learning (RL), which defines the long-term evaluation of a policy in a given state. In this paper, we propose Policy-extended Value Function Approximator (PeVFA) which extends the conventional value to be not only a function of state but also an explicit policy representation. Such an extension enables PeVFA to preserve values of multiple policies in contrast to a conventional one with limited capacity for only one policy, inducing the new characteristic of value generalization among policies. From both the theoretical and empirical lens, we study value generalization along the policy improvement path (called local generalization), from which we derive a new form of Generalized Policy Iteration with PeVFA to improve the conventional learning process. Besides, we propose a framework to learn the representation of an RL policy, studying several different approaches to learn an effective policy representation from policy network parameters and state-action pairs through contrastive learning and action prediction. In our experiments, Proximal Policy Optimization (PPO) with PeVFA significantly outperforms its vanilla counterpart in MuJoCo continuous control tasks, demonstrating the effectiveness of value generalization offered by PeVFA and policy representation learning.

show abstract

“…This can be viewed as a special two-agent cooperative game where the high level and low level learn to coordinate at the optimal hybrid actions. Although the hierarchical structure seems to be natural, it suffers from the high-level non-stationarity caused by offpolicy learning dynamics (Wang et al 2020), i.e., a discrete action can no longer induce the same transition in historical experiences due to the change of the low-level policy. All the above works focus on policy learning over original hybrid action space.…”

Section: Introductionmentioning

confidence: 99%

HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation

Tang¹,

Zheng²,

Hao³

et al. 2021

Preprint

View full text Add to dashboard Cite

Discrete-continuous hybrid action space is a natural setting in many practical problems, such as robot control and game AI. However, most previous Reinforcement Learning (RL) works only demonstrate the success in controlling with either discrete or continuous action space, while seldom take into account the hybrid action space. One naive way to address hybrid action RL is to convert the hybrid action space into a unified homogeneous action space by discretization or continualization, so that conventional RL algorithms can be applied. However, this ignores the underlying structure of hybrid action space and also induces the scalability issue and additional approximation difficulties, thus leading to degenerated results. In this paper, we propose Hybrid Action Representation (HyAR) to learn a compact and decodable latent representation space for the original hybrid action space. HyAR constructs the latent space and embeds the dependence between discrete action and continuous parameter via an embedding table and conditional Variantional Auto-Encoder (VAE). To further improve the effectiveness, the action representation is trained to be semantically smooth through unsupervised environmental dynamics prediction. Finally, the agent then learns its policy with conventional DRL algorithms in the learned representation space and interacts with the environment by decoding the hybrid action embeddings to the original action space. We evaluate HyAR in a variety of environments with discrete-continuous action space. The results demonstrate the superiority of HyAR when compared with previous baselines, especially for high-dimensional action spaces.

show abstract

I²HRL: Interactive Influence-based Hierarchical Reinforcement Learning

Cited by 11 publications

References 0 publications

Hierarchical Reinforcement Learning with Adversarially Guided Subgoals

Hierarchical Reinforcement Learning with Adversarially Guided Subgoals

What About Inputing Policy in Value Function: Policy Representation and Policy-extended Value Function Approximator

HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation

Contact Info

Product

Resources

About