Separated Proportional-Integral Lagrangian for Chance Constrained Reinforcement Learning

Peng, Baiyu; Mu, Yao; Duan, Jingliang; Guan, Yang; Li, Shengbo Eben; Chen, Jianyu

doi:10.1109/iv48863.2021.9575205

Cited by 9 publications

(4 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For our SAC integration, the gradient of the multiplicative value function already naturally results in something similar to a Lagrange multiplier. Commonly, the safety constraint is formulated as a constraint budget over the expected safety cost, e.g., [10], [20], [21], [23], [12], [11]. However, only considering the expected safety at each time step can cause the realized cost to exceed the constraint budget [22].…”

Section: Related Workmentioning

confidence: 99%

“…We tackle this issue by using reachability analysis and imposing zero constraint violation probability in our experiments. Another line of research improves the stability of the learning process by using derivatives and integrals of the constraint function yielding a PID approach [12], [23]. Similarly, our multiplicative value function improves stability by simplifying the learning task.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

A Multiplicative Value Function for Safe and Efficient Reinforcement Learning

Bührer,

Zhang,

Liniger

et al. 2023

2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

An emerging field of sequential decision problems is safe Reinforcement Learning (RL), where the objective is to maximize the reward while obeying safety constraints. Being able to handle constraints is essential for deploying RL agents in real-world environments, where constraint violations can harm the agent and the environment. To this end, we propose a safe model-free RL algorithm with a novel multiplicative value function consisting of a safety critic and a reward critic. The safety critic predicts the probability of constraint violation and discounts the reward critic that only estimates constraint-free returns. By splitting responsibilities, we facilitate the learning task leading to increased sample efficiency. We integrate our approach into two popular RL algorithms, Proximal Policy Optimization and Soft Actor-Critic, and evaluate our method in four safety-focused environments, including classical RL benchmarks augmented with safety constraints and robot navigation tasks with images and raw Lidar scans as observations. Finally, we make the zero-shot sim-to-real transfer where a differential drive robot has to navigate through a cluttered room. Our code can be found at https://github.com/nikeke19/ Safe-Mult-RL.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

A Multiplicative Value Function for Safe and Efficient Reinforcement Learning

Bührer,

Zhang,

Liniger

et al. 2023

2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

show abstract

“…As an alternative to risk metrics, previous works have directly bounded the probability of a system violating a constraint, which are known as chance constraints. They have been used for MDPs [23], partially observable Markov decision processes (POMDPs) [24] and RL [25]. However, these approaches do not account for how robustly a controller satisfies its constraints as we do in this paper.…”

Section: Literature Reviewmentioning

confidence: 99%

Risk verification of stochastic systems with neural network controllers

Cleaveland

Lindemann

Ivanov

et al. 2022

Artificial Intelligence

View full text Add to dashboard Cite

“…In this sense, developing a robust motion planning methodology is meaningful for stochastic multi-agent systems, such that the agents exhibits good robustness properties in the presence of other moving agents and static obstacles. One effective method is to characterize the uncertainties in a probabilistic manner and find the optimal sequence of control inputs subject to chance constraints [20], [21]. Thus, a risk bound can be stipulated in the chance constraints.…”

Section: Introductionmentioning

confidence: 99%

Velocity Obstacle Based Risk-Bounded Motion Planning for Stochastic Multi-Agent Systems

Zhang¹,

Ma²,

Cheng³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we present an innovative risk-bounded motion planning methodology for stochastic multi-agent systems. For this methodology, the disturbance, noise, and model uncertainty are considered; and a velocity obstacle method is utilized to formulate the collision-avoidance constraints in the velocity space.With the exploitation of geometric information of static obstacles and velocity obstacles, a distributed optimization problem with probabilistic chance constraints is formulated for the stochastic multi-agent system. Consequently, collision-free trajectories are generated under a prescribed collision risk bound. Due to the existence of probabilistic and disjunctive constraints, the distributed chance-constrained optimization problem is reformulated as a mixed-integer program by introducing the binary variable to improve computational efficiency. This approach thus renders it possible to execute the motion planning task in the velocity space instead of the position space, which leads to smoother collision-free trajectories for multi-agent systems and higher computational efficiency. Moreover, the risk of potential collisions is bounded with this robust motion planning methodology. To validate the effectiveness of the methodology, different scenarios for multiple agents are investigated, and the simulation results clearly show that the proposed approach can generate highquality trajectories under a predefined collision risk bound and avoid potential collisions effectively in the velocity space.

show abstract

Separated Proportional-Integral Lagrangian for Chance Constrained Reinforcement Learning

Cited by 9 publications

References 15 publications

A Multiplicative Value Function for Safe and Efficient Reinforcement Learning

A Multiplicative Value Function for Safe and Efficient Reinforcement Learning

Risk verification of stochastic systems with neural network controllers

Velocity Obstacle Based Risk-Bounded Motion Planning for Stochastic Multi-Agent Systems

Contact Info

Product

Resources

About