Invariant Transform Experience Replay: Data Augmentation for Deep Reinforcement Learning

Lin, Yijiong; Huang, Jiancong; Zimmer, Matthieu; Guan, Yisheng; Rojas, Juan; Weng, Paul

doi:10.1109/lra.2020.3013937

Cited by 25 publications

(14 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, it can be concluded that DDPG-NNFTSMC proposed in this paper has excellent control performance, and theoretically has good application prospects for industrial manipulators with uncertain dynamic models and general second-order nonlinear systems. Next, based on the research of scholars [12,31], we will try to apply DDPG-NNFTSMC to other different scenarios and combine it with deep reinforcement learning to further improve the control performance [32,36,37].…”

Section: Discussionmentioning

confidence: 99%

Nonlinear Nonsingular Fast Terminal Sliding Mode Control Using Deep Deterministic Policy Gradient

Huang

et al. 2021

Applied Sciences

View full text Add to dashboard Cite

Background: As a control strategy of industrial robots, sliding mode control has the advantages of fast response and simple physical implementation, but it still has the problems of chattering and low tracking accuracy caused by chattering. This paper proposes a new sliding mode control strategy for the application of industrial robot control, which effectively solves these problems. Methods: In this paper, a deep deterministic policy gradient–nonlinear nonsingular fast terminal sliding mode control (DDPG–NNFTSMC) strategy is proposed for industrial robot control. In order to improve the tracking control accuracy and anti-interference ability, DDPG is used to approach the uncertainties of the system in real time, which ensures the robustness of the system in various uncertain environments. Lyapunov function is used to prove the stability and finite time convergence of the system. Compared with the nonsingular terminal sliding mode control (NTSMC), the time to reach the equilibrium point is shorter. With the help of MATLAB/Simulink, the tracking accuracy and control effects are compared with traditional terminal sliding mode control (TSMC), NTSMC and radial basis function–sliding mode control (RBF–SMC), the results showed that it had the advantages of nonsingularity, finite time convergence, small tracking error. The motion accuracy and anti-interference ability of the uncertain manipulator system was further improved, and the chattering problem of the system in the motion process is effectively eliminated.

show abstract

Section: Discussionmentioning

confidence: 99%

Nonlinear Nonsingular Fast Terminal Sliding Mode Control Using Deep Deterministic Policy Gradient

Huang

et al. 2021

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…Interestingly, the value of approximate symmetries of MDPs has been explored in some theoretical work [45,50], but without architectures that can make use of it. Additionally, data augmentation, while not able to bake in architectural equivariance, has been successfully applied to encouraging equivariance on image tasks [30] and recently even on tabular state vectors [33,40].…”

Section: Related Workmentioning

confidence: 99%

Residual Pathway Priors for Soft Equivariance Constraints

Finzi¹,

Benton²,

Wilson³

2021

Preprint

View full text Add to dashboard Cite

There is often a trade-off between building deep learning systems that are expressive enough to capture the nuances of the reality, and having the right inductive biases for efficient learning. We introduce Residual Pathway Priors (RPPs) as a method for converting hard architectural constraints into soft priors, guiding models towards structured solutions, while retaining the ability to capture additional complexity. Using RPPs, we construct neural network priors with inductive biases for equivariances, but without limiting flexibility. We show that RPPs are resilient to approximate or misspecified symmetries, and are as effective as fully constrained models even when symmetries are exact. We showcase the broad applicability of RPPs with dynamical systems, tabular data, and reinforcement learning. In Mujoco locomotion tasks, where contact forces and directional rewards violate strict equivariance assumptions, the RPP outperforms baseline model-free RL agents, and also improves the learned transition models for model-based RL.

show abstract

“…Recent work on symmetries in single agent deep reinforcement learning has shown improvements in terms of data effiency. Such work revolves around symmetries in policy networks (van der Pol et al, 2020;Simm et al, 2020), symmetric filters (Clark & Storkey, 2014), invariant data augmentation (Laskin et al, 2020;Kostrikov et al, 2020) or equivariant trajectory augmentation (Lin et al, 2020;Mavalankar, 2020;Mishra et al, 2019) These approaches are only suitable for single agent problems or centralized multi-agent controllers. Here, we solve the problem of enforcing global equivariance while still allowing distributed execution.…”

Section: Related Workmentioning

confidence: 99%

“…Equivariant and geometric deep learning have gained traction in recent years, showing promising results in supervised learning (Cohen & Welling, 2016;Winkels & Cohen, 2018;Weiler et al, 2018;Weiler & Cesa, 2019;Worrall et al, 2017;Fuchs et al, 2020;Thomas et al, 2018), unsupervised learning (Dey et al, 2021) and reinforcement learning (van der Pol et al, 2020;Simm et al, 2020). In single agent reinforcement learning, enforcing equivariance to group symmetries has been shown to improve data efficiency, for example with MDP homomorphic networks (van der Pol et al, 2020), trajectory augmentation (Lin et al, 2020;Mavalankar, 2020), or symmetric locomotion policies (Abdolhosseini et al, 2019). Equivariant approaches enable a single agent to learn policies more efficiently within its environment by sharing weights between state-action pairs that are equivalent under a transformation.…”

Section: Introductionmentioning

confidence: 99%

Multi-Agent MDP Homomorphic Networks

Pol¹,

Hoof²,

Oliehoek³

et al. 2021

Preprint

View full text Add to dashboard Cite

This paper introduces Multi-Agent MDP Homomorphic Networks, a class of networks that allows distributed execution using only local information, yet is able to share experience between global symmetries in the joint state-action space of cooperative multi-agent systems. In cooperative multi-agent systems, complex symmetries arise between different configurations of the agents and their local observations. For example, consider a group of agents navigating: rotating the state globally results in a permutation of the optimal joint policy. Existing work on symmetries in single agent reinforcement learning can only be generalized to the fully centralized setting, because such approaches rely on the global symmetry in the full state-action spaces, and these can result in correspondences across agents. To encode such symmetries while still allowing distributed execution we propose a factorization that decomposes global symmetries into local transformations. Our proposed factorization allows for distributing the computation that enforces global symmetries over local agents and local interactions. We introduce a multi-agent equivariant policy network based on this factorization. We show empirically on symmetric multi-agent problems that distributed execution of globally symmetric policies improves data efficiency compared to non-equivariant baselines.

show abstract

Invariant Transform Experience Replay: Data Augmentation for Deep Reinforcement Learning

Cited by 25 publications

References 12 publications

Nonlinear Nonsingular Fast Terminal Sliding Mode Control Using Deep Deterministic Policy Gradient

Nonlinear Nonsingular Fast Terminal Sliding Mode Control Using Deep Deterministic Policy Gradient

Residual Pathway Priors for Soft Equivariance Constraints

Multi-Agent MDP Homomorphic Networks

Contact Info

Product

Resources

About