Model-Based Actor-Critic with Chance Constraint for Stochastic System

Improve Generalization of Driving Policy at Signalized Intersections with Adversarial Learning

“…Note that this choice procedure is conducted at each step of the driving process, therefore the output mode will provably be changing with the state of vehicle and light duration. Besides, the stop or pass mode only determines the speed error ∆v in the track error of (12). After that, each candidate route will be further adopted to construct other elements ∆x, ∆y, ∆ϕ respectively and V w * will select the optimal one to track.…”

“…al (2020) considered a risk function and bounded the expected risk within predefined hard constraints, showing that this could assure the driving safety at the simple two-vehicle interacting at an intersection [10]. Besides, some other works also have explored different constraints formulation for driving tasks such as control barrier function [11], chance constraint [12] and constraints in continuous time [13]. Inspired by these constraint optimizations, Guan et.al (2021) recently proposed the integrated decision and control (IDC) framework for autonomous driving which intuitively adopted RL to solve the constrained optimal control problems (COCP) based on multiple candidate paths generated by static path planner [14].…”

Separated Proportional-Integral Lagrangian for Chance Constrained Reinforcement Learning

“…However, it turns out that the gradient ∇ θ p s (π) is rather difficult to compute, which is also a major challenge in chance constrained problems [20], [23]. Previous researchers in chance constrained RL usually replace ∇ θ p s (π) with the gradient of a lower bound of p s without sufficient theoretical guarantees [14], [15]. In this paper, we introduce an analytical approximated gradient with theoretical basis [23].…”

“…Recently, some RL researchers begin to investigate including different forms of safety constraints in RL algorithms to improve safety for real-world applications [10]- [13]. One of the most popular forms is the chance constraint, which constrains the possibility of the control policy violating the state constraint below a given level [10], [14], [15]. Chance constraint gives an intuitive and quantitative measure of the safety level of the control policy, so it is suitable to represent the safety demands in real-world systems with uncertainty.…”

“…Existing strategies used to solve the chance constrained RL problems can be roughly categorized into two approaches. The first solution is the penalty method that gives a large penalty to the objective function for violation of the constraint [9], [14]. Although this approach is very straightforward and simple to implement, it requires the penalty weight to strike a balance between safety and performance correctly.…”