Global Convergence of Policy Gradient Primal-dual Methods for Risk-constrained LQRs

Zhao, Feiran; You, Keyou; Başar, Tamer

doi:10.48550/arxiv.2104.04901

Cited by 4 publications

(3 citation statements)

References 36 publications

(71 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Remark 2. It is worth mentioning that PO has also been investigated in control problems with robustness and risk-sensitivity concerns other than the LEQG/H ∞ settings discussed in this survey (see 50,52,55,115,[124][125][126][127][128][129].…”

Section: For Anymentioning

confidence: 99%

Toward a Theoretical Foundation of Policy Optimization for Learning Control Policies

Zhang

et al. 2023

Annu. Rev. Control Robot. Auton. Syst.

View full text Add to dashboard Cite

Gradient-based methods have been widely used for system design and optimization in diverse application domains. Recently, there has been a renewed interest in studying theoretical properties of these methods in the context of control and reinforcement learning. This article surveys some of the recent developments on policy optimization, a gradient-based iterative approach for feedback control synthesis that has been popularized by successes of reinforcement learning. We take an interdisciplinary perspective in our exposition that connects control theory, reinforcement learning, and large-scale optimization. We review a number of recently developed theoretical results on the optimization landscape, global convergence, and sample complexityof gradient-based methods for various continuous control problems, such as the linear quadratic regulator (LQR), [Formula: see text] control, risk-sensitive control, linear quadratic Gaussian (LQG) control, and output feedback synthesis. In conjunction with these optimization results, we also discuss how direct policy optimization handles stability and robustness concerns in learning-based control, two main desiderata in control engineering. We conclude the survey by pointing out several challenges and opportunities at the intersection of learning and control.

show abstract

Section: For Anymentioning

confidence: 99%

Toward a Theoretical Foundation of Policy Optimization for Learning Control Policies

Zhang

et al. 2023

Annu. Rev. Control Robot. Auton. Syst.

View full text Add to dashboard Cite

show abstract

“…The results of [43] were extended to the infinite horizon case in [44]. The performance of the policy gradient algorithm in the case of risk-constrained Linear Quadratic Regulators was also studied in [45]. Predictive variance constraints have also been used as a measure of risk in portfolio optimization [46]; different from our paper, the noise is limited to Gaussian distributions and the variance is with respect to linear stage costs.…”

Section: Related Workmentioning

confidence: 99%

Linear Quadratic Control with Risk Constraints

Tsiamis¹,

Kalogerias²,

Ribeiro³

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose a new risk-constrained formulation of the classical Linear Quadratic (LQ) stochastic control problem for general partially-observed systems. Our framework is motivated by the fact that the risk-neutral LQ controllers, although optimal in expectation, might be ineffective under relatively infrequent, yet statistically significant extreme events. To effectively trade between average and extreme event performance, we introduce a new risk constraint, which explicitly restricts the total expected predictive variance of the state penalty by a user-prescribed level. We show that, under certain conditions on the process noise, the optimal risk-aware controller can be evaluated explicitly and in closed form. In fact, it is affine relative to the minimum mean square error (mmse) state estimate. The affine term pushes the state away from directions where the noise exhibits heavy tails, by exploiting the third-order moment (skewness) of the noise. The linear term regulates the state more strictly in riskier directions, where both the prediction error (conditional) covariance and the state penalty are simultaneously large; this is achieved by inflating the state penalty within a new filtered Riccati difference equation. We also prove that the new risk-aware controller is internally stable, regardless of parameter tuning, in the special cases of i) fully-observed systems, and ii) partially-observed systems with Gaussian noise. The properties of the proposed risk-aware LQ framework are lastly illustrated via indicative numerical examples.

show abstract

“…This point of view was initiated when the LQR cost was shown to be gradient dominant [19], facilitating a global convergence guarantee of first order methods for this problem-despite of its non-convexity. Since then, PO using first order methods has been investigated for variants of LQR problem, such as OLQR [20], model-free setup [21], and riskconstrained LQR [22] just to name a few. The gradient dominant property, however, is only known to be valid with respect to the global optimum of the unconstrained case, and is not necessarily expected for the general constrained LQR problems.…”

Section: Introductionmentioning

confidence: 99%

Policy Optimization over Submanifolds for Constrained Feedback Synthesis

Talebi¹,

Mesbahi²

2022

Preprint

View full text Add to dashboard Cite

In this paper, we study linearly constrained policy optimizations over the manifold of Schur stabilizing controllers, equipped with a Riemannian metric that emerges naturally in the context of optimal control problems.We provide extrinsic analysis of a generic constrained smooth cost function, that subsequently facilitates subsuming any such constrained problem into this framework. By studying the second order geometry of this manifold, we provide a Newton-type algorithm with local convergence guarantees that exploits this inherent geometry without relying on the exponential mapping nor a retraction. The algorithm hinges instead upon the developed stability certificate and the linear structure of the constraints. We then apply our methodology to two well-known constrained optimal control problems. Finally, several numerical examples showcase the performance of the proposed algorithm.

show abstract

Global Convergence of Policy Gradient Primal-dual Methods for Risk-constrained LQRs

Cited by 4 publications

References 36 publications

Toward a Theoretical Foundation of Policy Optimization for Learning Control Policies

Toward a Theoretical Foundation of Policy Optimization for Learning Control Policies

Linear Quadratic Control with Risk Constraints

Policy Optimization over Submanifolds for Constrained Feedback Synthesis

Contact Info

Product

Resources

About