Model-Free $μ$ Synthesis via Adversarial Reinforcement Learning

Keivan, Darioush; Havens, Aaron; Seiler, Peter; Dullerud, Geir E.; Hu, Bin

doi:10.48550/arxiv.2111.15537

Cited by 3 publications

(3 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An important future task is to extend the PGD attack for input-output gain analysis which is crucial for robust control. Recently, the H ∞ input-output gain has been used in robust reinforcement learning [31]- [36]. A general-purpose input-output gain analysis will play a crucial role for the developments of robust DRL methods.…”

Section: Concluding Remarks and Future Workmentioning

confidence: 99%

Revisiting PGD Attacks for Stability Analysis of High-Dimensional Nonlinear Systems and Perception-Based Control

Havens

Kevian

Seiler

et al. 2023

IEEE Control Syst. Lett.

View full text Add to dashboard Cite

Many existing region-of-attraction (ROA) analysis tools find difficulty in addressing feedback systems with large-scale neural network (NN) policies and/or highdimensional sensing modalities such as cameras. In this letter, we tailor the projected gradient descent (PGD) attack method as a general-purpose ROA analysis tool for highdimensional nonlinear systems and end-to-end perceptionbased control. We show that the ROA analysis can be approximated as a constrained maximization problem such that PGD-based iterative methods can be directly applied. In the model-based setting, we show that the PGD updates can be efficiently performed using back-propagation. In the model-free setting, we propose a finite-difference PGD estimate which is general and only requires a black-box simulator for generating the trajectories of the closed-loop system given any initial state. Finally, we demonstrate the scalability and generality of our analysis tool on several numerical examples with large state dimensions or complex image observations.

show abstract

Section: Concluding Remarks and Future Workmentioning

confidence: 99%

Revisiting PGD Attacks for Stability Analysis of High-Dimensional Nonlinear Systems and Perception-Based Control

Havens

Kevian

Seiler

et al. 2023

IEEE Control Syst. Lett.

View full text Add to dashboard Cite

show abstract

“…Recently, the H ∞ input-output gain has been used in robust reinforcement learning (Han et al, 2019;Zhang et al, 2020a,b;Donti et al, 2020;Zhang et al, 2021). It has been shown that an efficient input-output gain analysis can be combined with adversarial reinforcement learning (Pinto et al, 2017) to improve the robustness in the linear control setting (Keivan et al, 2021). A general-purpose input-output gain analysis will play a crucial role for the developments of robust DRL in the nonlinear or perception-based setting.…”

Section: Concluding Remarks and Future Workmentioning

confidence: 99%

Revisiting PGD Attacks for Stability Analysis of Large-Scale Nonlinear Systems and Perception-Based Control

Havens¹,

Keivan²,

Seiler³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Many existing region-of-attraction (ROA) analysis tools find difficulty in addressing feedback systems with large-scale neural network (NN) policies and/or high-dimensional sensing modalities such as cameras. In this paper, we tailor the projected gradient descent (PGD) attack method developed in the adversarial learning community as a general-purpose ROA analysis tool for large-scale nonlinear systems and end-to-end perception-based control. We show that the ROA analysis can be approximated as a constrained maximization problem whose goal is to find the worst-case initial condition which shifts the terminal state the most. Then we present two PGD-based iterative methods which can be used to solve the resultant constrained maximization problem. Our analysis is not based on Lyapunov theory, and hence requires minimum information of the problem structures. In the model-based setting, we show that the PGD updates can be efficiently performed using backpropagation. In the model-free setting (which is more relevant to ROA analysis of perception-based control), we propose a finite-difference PGD estimate which is general and only requires a blackbox simulator for generating the trajectories of the closed-loop system given any initial state. We demonstrate the scalability and generality of our analysis tool on several numerical examples with large-scale NN policies and high-dimensional image observations. We believe that our proposed analysis serves as a meaningful initial step toward further understanding of closed-loop stability of large-scale nonlinear systems and perception-based control.

show abstract

“…In this work, we extend the nonsmooth optimization perspective on µ-synthesis (Apkarian, 2011) to the model-free setting. Notice that the model-free state-feedback µ-synthesis has been previously addressed via combining DK-iteration and a central-path algorithm that adopts robust adversarial reinforcement learning (RARL) as subroutines for finding analytical center in the K step (Keivan et al, 2021). However, such an approach is not directly applicable in the general output-feedback setting.…”

Section: Introductionmentioning

confidence: 99%

Model-Free μ Synthesis via Adversarial Reinforcement Learning

Keivan

Havens

Seiler

et al. 2022

2022 American Control Conference (ACC)

View full text Add to dashboard Cite

In this paper, we revisit model-free policy search on an important robust control benchmark, namely µ synthesis. In the general output-feedback setting, there do not exist convex formulations for this problem, and hence global optimality guarantees are not expected. Apkarian (2011) presented a nonconvex nonsmooth policy optimization approach for this problem, and achieved state-of-the-art design results via using subgradient-based policy search algorithms which generate update directions in a model-based manner. Despite the lack of convexity and global optimality guarantees, these subgradient-based policy search methods have led to impressive numerical results in practice. Built upon such a policy optimization persepctive, our paper extends these subgradient-based search methods to a model-free setting. Specifically, we examine the effectiveness of two model-free policy optimization strategies: the model-free non-derivative sampling method and the zeroth-order policy search with uniform smoothing. We performed an extensive numerical study to demonstrate that both methods consistently replicate the design outcomes achieved by their model-based counterparts. Additionally, we provide some theoretical justifications showing that convergence guarantees to stationary points can be established for our model-free µ-synthesis under some assumptions related to the coerciveness of the cost function. Overall, our results demonstrate that derivative-free policy optimization offers a competitive and viable approach for solving general output-feedback µ-synthesis problems in the model-free setting.

show abstract

Model-Free $μ$ Synthesis via Adversarial Reinforcement Learning

Cited by 3 publications

References 35 publications

Revisiting PGD Attacks for Stability Analysis of High-Dimensional Nonlinear Systems and Perception-Based Control

Revisiting PGD Attacks for Stability Analysis of High-Dimensional Nonlinear Systems and Perception-Based Control

Revisiting PGD Attacks for Stability Analysis of Large-Scale Nonlinear Systems and Perception-Based Control

Model-Free μ Synthesis via Adversarial Reinforcement Learning

Contact Info

Product

Resources

About