Learning to Walk in the Real World with Minimal Human Effort

Ha, Sehoon; Xu, Peng; Tan, Zhenyu; Levine, Sergey; Tan, Jie

doi:10.48550/arxiv.2002.08550

Cited by 28 publications

(39 citation statements)

References 32 publications

(46 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…] is an entropy of the stochastic policy 𝜋 and 𝛼 ≥ 0 is an entropy temperature. Furthermore, we use the Lagrangian relaxation method [22], [23], [24] to solve the 𝜏-CMDP problem.…”

Section: Dmentioning

confidence: 99%

Deep reinforcement learning under signal temporal logic constraints using Lagrangian relaxation

Ikemoto¹,

Ushio²

2022

Preprint

View full text Add to dashboard Cite

Deep reinforcement learning (DRL) has attracted much attention as an approach to solve sequential decision making problems without mathematical models of systems or environments. In general, a constraint may be imposed on the decision making. In this study, we consider the optimal decision making problems with constraints to complete temporal highlevel tasks in the continuous state-action domain. We describe the constraints using signal temporal logic (STL), which is useful for time sensitive control tasks since it can specify continuous signals within a bounded time interval. To deal with the STL constraints, we introduce an extended constrained Markov decision process (CMDP), which is called a 𝜏-CMDP. We formulate the STL constrained optimal decision making problem as the 𝜏-CMDP and propose a two-phase constrained DRL algorithm using the Lagrangian relaxation method. Through simulations, we also demonstrate the learning performance of the proposed algorithm.

show abstract

Section: Dmentioning

confidence: 99%

Deep reinforcement learning under signal temporal logic constraints using Lagrangian relaxation

Ikemoto¹,

Ushio²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Reset-free RL has been studied by previous works with a focus on safety (Eysenbach et al, 2017), automated and unattended learning in the real world (Han et al, 2015;Zhu et al, 2020;, skill discovery Lu et al, 2020), and providing a curriculum (Sharma et al, 2021). Strategies to learn reset-free behavior include directly learning a backward reset controller (Eysenbach et al, 2017), learning a set of auxillary tasks that can serve as an approximate reset (Ha et al, 2020;, or using a novelty seeking reset controller (Zhu et al, 2020). Complementary to this literature, we aim to develop a set of benchmarks and a framework that allows for this class of algorithms to be studied in a unified way.…”

Section: Related Workmentioning

confidence: 99%

Autonomous Reinforcement Learning: Formalism and Benchmarking

Sharma¹,

Xu²,

Sardana³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Reinforcement learning (RL) provides a naturalistic framing for learning through trial and error, which is appealing both because of its simplicity and effectiveness and because of its resemblance to how humans and animals acquire skills through experience. However, real-world embodied learning, such as that performed by humans and animals, is situated in a continual, non-episodic world, whereas common benchmark tasks in RL are episodic, with the environment resetting between trials to provide the agent with multiple attempts. This discrepancy presents a major challenge when attempting to take RL algorithms developed for episodic simulated environments and run them on real-world platforms, such as robots. In this paper, we aim to address this discrepancy by laying out a framework for Autonomous Reinforcement Learning (ARL): reinforcement learning where the agent not only learns through its own experience, but also contends with lack of human supervision to reset between trials. We introduce a simulated benchmark EARL 1 around this framework, containing a set of diverse and challenging simulated tasks reflective of the hurdles introduced to learning when only a minimal reliance on extrinsic intervention can be assumed. We show that standard approaches to episodic RL and existing approaches struggle as interventions are minimized, underscoring the need for developing new algorithms for reinforcement learning with a greater focus on autonomy.

show abstract

“…The learned policy has been successfully deployed on real robots by using domain randomization (DR) [3], system identification [5], or real-world adaptation [18]. Alternatively, researchers [4], [19] have investigated learning policies directly from realworld experience, which can intrinsically overcome sim-toreal gaps. Sample efficiency is a critical challenge for deep RL approaches, which can be improved by leveraging modelbased control strategies [17], [20], [21].…”

Section: Related Workmentioning

confidence: 99%

Solving Challenging Control Problems Using Two-Staged Deep Reinforcement Learning

Sontakke

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We present a two-staged deep reinforcement learning algorithm for solving challenging control problems. Deep reinforcement learning (deep RL) has been an effective tool for solving many high-dimensional continuous control problems, but it cannot effectively solve challenging problems with certain properties, such as sparse reward functions or sensitive dynamics. In this work, we propose an approach that decomposes the given problem into two stages: motion planning and motion imitation. The motion planning stage seeks to compute a feasible motion plan with approximated dynamics by directly sampling the state space rather than exploring random control signals.Once the motion plan is obtained, the motion imitation stage learns a control policy that can imitate the given motion plan with realistic sensors and actuations. We demonstrate that our approach can solve challenging control problemsrocket navigation and quadrupedal locomotion -which cannot be solved by the standard MDP formulation. The supplemental video can be found at: https://youtu.be/FYLo1Ov_8-g

show abstract

Learning to Walk in the Real World with Minimal Human Effort

Cited by 28 publications

References 32 publications

Deep reinforcement learning under signal temporal logic constraints using Lagrangian relaxation

Deep reinforcement learning under signal temporal logic constraints using Lagrangian relaxation

Autonomous Reinforcement Learning: Formalism and Benchmarking

Solving Challenging Control Problems Using Two-Staged Deep Reinforcement Learning

Contact Info

Product

Resources

About