On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Control

Bedi, Amrit Singh; Parayil, Anjaly; Zhang, Junyu; Wang, Mengdi; Koppel, Alec

doi:10.48550/arxiv.2106.08414

Cited by 2 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Heavy-Tailed Policy Parametrization: The idea of parametrizing policies via heavy-tailed distribution has appeared in the reinforcement learning literature [23], [37]. The authors in [37] proposed to utilize beta distribution for policy parametrization but are restricted to dense reward structure environments.…”

Section: Learning From Demonstrationmentioning

confidence: 99%

“…The authors in [37] proposed to utilize beta distribution for policy parametrization but are restricted to dense reward structure environments. Authors in [23] have focused on the development of heavy-tailed policy gradient to avoid convergence to local maxima and do not explicitly deal with sparse rewards. This work focus on sparse reward continuous control environments and extensive experimental evaluations to support the importance of heavy-tailed policy parametrization.…”

Section: Learning From Demonstrationmentioning

confidence: 99%

“…This ensures that the probability of taking any action in a given state is always non-zero, which would help to visit the complete state space and look for rewards. Recently, authors in [23] have extended the idea of persistent exploration to continuous spaces and have proposed to utilize heavy-tailed policies to avoid convergence of policy gradient methods to spurious local maximas. Taking motivation from [23], we ask the following question "Can heavy-tailed policies make model-free RL sample efficient for practical robotics tasks that involve sparse reward structure without any expert demonstrations?…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Dealing with Sparse Rewards in Continuous Control Robotics via Heavy-Tailed Policies

Chakraborty¹,

Bedi²,

Koppel³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we present a novel Heavy-Tailed Stochastic Policy Gradient (HT-PSG) algorithm to deal with the challenges of sparse rewards in continuous control problems. Sparse reward is common in continuous control robotics tasks such as manipulation and navigation, and makes the learning problem hard due to non-trivial estimation of value functions over the state space. This demands either reward shaping or expert demonstrations for the sparse reward environment. However, obtaining high-quality demonstrations is quite expensive and sometimes even impossible. We propose a heavytailed policy parametrization along with a modified momentumbased policy gradient tracking scheme (HT-SPG) to induce a stable exploratory behavior to the algorithm. The proposed algorithm does not require access to expert demonstrations. We test the performance of HT-SPG on various benchmark tasks of continuous control with sparse rewards such as 1D Mario, Pathological Mountain Car, Sparse Pendulum in OpenAI Gym, and Sparse MuJoCo environments (Hopper-v2). We show consistent performance improvement across all tasks in terms of high average cumulative reward. HT-SPG also demonstrates improved convergence speed with minimum samples, thereby emphasizing the sample efficiency of our proposed algorithm.

show abstract

Section: Learning From Demonstrationmentioning

confidence: 99%

Section: Learning From Demonstrationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Dealing with Sparse Rewards in Continuous Control Robotics via Heavy-Tailed Policies

Chakraborty¹,

Bedi²,

Koppel³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…But the major challenge is how to induce inherent exploration into the training without reshaping the rewards. Recent work in literature [20,21] suggest that one way to handle these is to use heavy-tailed policy (such as Cauchy) parametrization techniques. Motivated by these factors, in this work, we try to find optimal behaviors for outdoor navigation tasks while directly operating under sparse reward settings.…”

Section: Introductionmentioning

confidence: 99%

HTRON:Efficient Outdoor Navigation with Sparse Rewards via Heavy Tailed Adaptive Reinforce Algorithm

Weerakoon¹,

Chakraborty²,

Karapetyan³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

We present a novel approach to improve the performance of deep reinforcement learning (DRL) based outdoor robot navigation systems. Most, existing DRL methods are based on carefully designed dense reward functions that learn the efficient behavior in an environment. We circumvent this issue by working only with sparse rewards (which are easy to design), and propose a novel adaptive Heavy-Tailed Reinforce algorithm for Outdoor Navigation called HTRON. Our main idea is to utilize heavy-tailed policy parametrizations which implicitly induce exploration in sparse reward settings. We evaluate the performance of HTRON against Reinforce, PPO and TRPO algorithms in three different outdoor scenarios: goal-reaching, obstacle avoidance, and uneven terrain navigation. We observe in average an increase of 34.41% in terms of success rate, 15.15% decrease in the average time steps taken to reach the goal, and 24.9% decrease in the elevation cost compared to the navigation policies obtained by the other methods. Further, we demonstrate that our algorithm can be transferred directly into a Clearpath Husky robot to perform outdoor terrain navigation in real-world scenarios. Demonstration video is available at gamma.umd.edu/htron.

show abstract

On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Control

Cited by 2 publications

References 0 publications

Dealing with Sparse Rewards in Continuous Control Robotics via Heavy-Tailed Policies

Dealing with Sparse Rewards in Continuous Control Robotics via Heavy-Tailed Policies

HTRON:Efficient Outdoor Navigation with Sparse Rewards via Heavy Tailed Adaptive Reinforce Algorithm

Contact Info

Product

Resources

About