Model-Based Chance-Constrained Reinforcement Learning via Separated Proportional-Integral Lagrangian

Peng, Baiyu; Duan, Jingliang; Chen, Jianyu; Li, Shengbo Eben; Xie, Genjin; Congsheng, Zhang,; Guan, Yang; Mu, Yao; Sun, Enxin

doi:10.1109/tnnls.2022.3175595

Cited by 8 publications

(2 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Agrawal et al [48] represented constrained optimization problems as implicit layers in deep neural networks, leading to the introduction of neural network policy architectures designed to handle predefined constraints [49], [50]. Peng et al [51] proposed a novel proportional-integral Lagrangian method to handle hance constraints in RL setting. A conceptual idea of backpropagating through the learned system model parametrized via convex neural networks was investigated in [52].…”

Section: B Related Work 1) Learning-based Model Predictive Controlmentioning

confidence: 99%

Learning Constrained Parametric Differentiable Predictive Control Policies With Guarantees

Drgoňa,

Tuor,

Vrabie

2024

IEEE Trans. Syst. Man Cybern, Syst.

View full text Add to dashboard Cite

We present differentiable predictive control (DPC), a method for offline learning of constrained neural control policies for nonlinear dynamical systems with performance guarantees. We show that the sensitivities of the parametric optimal control problem can be used to obtain direct policy gradients. Specifically, we employ automatic differentiation (AD) to efficiently compute the sensitivities of the model predictive control (MPC) objective function and constraints penalties. To guarantee safety upon deployment, we derive probabilistic guarantees on closed-loop stability and constraint satisfaction based on indicator functions and Hoeffding's inequality. We empirically demonstrate that the proposed method can learn neural control policies for various parametric optimal control tasks. In particular, we show that the proposed DPC method can stabilize systems with unstable dynamics, track time-varying references, and satisfy nonlinear state and input constraints. Our DPC method has practical time savings compared to alternative approaches for fast and memoryefficient controller design. Specifically, DPC does not depend on a supervisory controller as opposed to approximate MPC based on imitation learning. We demonstrate that, without losing performance, DPC is scalable with greatly reduced demands on memory and computation compared to implicit and explicit MPC while being more sample efficient than model-free reinforcement learning (RL) algorithms.

show abstract

Section: B Related Work 1) Learning-based Model Predictive Controlmentioning

confidence: 99%

Learning Constrained Parametric Differentiable Predictive Control Policies With Guarantees

Drgoňa,

Tuor,

Vrabie

2024

IEEE Trans. Syst. Man Cybern, Syst.

View full text Add to dashboard Cite

show abstract

“…Chance-Constrained RL: Chance constrained RL has gained popularity in recent years. Some model-based chance-constrained RL methods improve the overconservative policy with efficient evaluations (Peng et al 2021(Peng et al , 2022. However model-based methods are challenging to deploy on real systems.…”

Section: Related Workmentioning

confidence: 99%

NENYA: Cascade Reinforcement Learning for Cost-Aware Failure Mitigation at Microsoft 365

Wang

Zhao

et al. 2022

Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

We address the challenge of learning safe and robust decision policies in presence of uncertainty in context of the real scientific problem of adaptive resource oversubscription to enhance resource efficiency while ensuring safety against resource congestion risk. Traditional supervised prediction or forecasting models are ineffective in learning adaptive policies whereas standard online optimization or reinforcement learning is difficult to deploy on real systems. Offline methods such as imitation learning (IL) are ideal since we can directly leverage historical resource usage telemetry. But, the underlying aleatoric uncertainty in such telemetry is a critical bottleneck. We solve this with our proposed novel chanceconstrained imitation learning framework, which ensures implicit safety against uncertainty in a principled manner via a combination of stochastic (chance) constraints on resource congestion risk and ensemble value functions. This leads to substantial (≈ 3 − 4×) improvement in resource efficiency and safety in many oversubscription scenarios, including resource management in cloud services.

show abstract

GOPS: A general optimal control problem solver for autonomous driving and industrial control applications

Wang¹,

Zhang²,

Gao³

et al. 2023

Communications in Transportation Research

View full text Add to dashboard Cite

Model-Based Chance-Constrained Reinforcement Learning via Separated Proportional-Integral Lagrangian

Cited by 8 publications

References 31 publications

Learning Constrained Parametric Differentiable Predictive Control Policies With Guarantees

Learning Constrained Parametric Differentiable Predictive Control Policies With Guarantees

NENYA: Cascade Reinforcement Learning for Cost-Aware Failure Mitigation at Microsoft 365

GOPS: A general optimal control problem solver for autonomous driving and industrial control applications

Contact Info

Product

Resources

About