On the Linear Convergence of Random Search for Discrete-Time LQR

Mohammadi, Hesameddin; Soltanolkotabi, Mahdi; Jovanović, Mihailo R.

doi:10.1109/lcsys.2020.3006256

Cited by 41 publications

(50 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We apply a two-point gradient estimation for ∇J(K), as it yields better sample complexity than the one-point setting. Motivated by Mohammadi et al (2020), we show that with a large probability, (3) converges linearly using only log(1/ ) simulation time and total sampled trajectories with a desired accuracy of the cost, which significantly improves the polynomial sample complexity in Perdomo et al (2021). By letting = J − J * γ i+1 , we have the following result.…”

Section: Resultsmentioning

confidence: 80%

“…with ∇J γ (K k ) estimated by Algorithm 2. The result (Mohammadi et al, 2020, Theorem 1) can be restated as follows. Lemma 11 significantly improves from the existing literature in both the simulation time and the number of sampled trajectories.…”

Section: Discussionmentioning

confidence: 96%

“…Finally, we briefly discuss the analysis of policy gradient methods for Algorithm 3. Since Mohammadi et al (2021Mohammadi et al ( , 2020 only consider the randomness of x 0 , their convergence analysis might not be able to applied to the stochastic noise setting. Also, assumes that the simulation time is infinite to obtain a unbiased estimation for the cost, we cannot directly use their results.…”

Section: Appendix a Policy Gradient Methods And Sample Complexitymentioning

confidence: 99%

See 2 more Smart Citations

Learning Stabilizing Controllers of Linear Systems via Discount Policy Gradient

Zhao¹,

Fu²,

You³

2021

Preprint

View full text Add to dashboard Cite

Stability is one of the most fundamental requirements for systems synthesis. In this paper, we address the stabilization problem for unknown linear systems via policy gradient (PG) methods. We leverage a key feature of PG for Linear Quadratic Regulator (LQR), i.e., it drives the policy away from the boundary of the unstabilizing region along the descent direction, provided with an initial policy with finite cost. To this end, we discount the LQR cost with a factor, by adaptively increasing which gradient leads the policy to the stabilizing set while maintaining a finite cost. Based on the Lyapunov theory, we design an update rule for the discount factor which can be directly computed from data, rendering our method purely model-free. Compared to recent work (Perdomo et al., 2021), our algorithm allows the policy to be updated only once for each discount factor. Moreover, the number of sampled trajectories and simulation time for gradient descent is significantly reduced to O(log(1/ )) for the desired accuracy . Finally, we conduct simulations on both small-scale and large-scale examples to show the efficiency of our discount PG method.

show abstract

Section: Resultsmentioning

confidence: 80%

Section: Discussionmentioning

confidence: 96%

Section: Appendix a Policy Gradient Methods And Sample Complexitymentioning

confidence: 99%

See 1 more Smart Citation

Learning Stabilizing Controllers of Linear Systems via Discount Policy Gradient

Zhao¹,

Fu²,

You³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Prior work: The majority of data-driven control approaches have focused on providing solutions to the linear quadratic regulator (LQR) problem. Among these works, we focus on the direct methods that do not require a system identification step (Hjalmarsson et al, 1998;Fazel et al, 2018;Mohammadi et al, 2020;Bradtke et al, 1994;De Persis and Tesi, 2019;Trentelman et al, 2020). Specifically, we highlight the work in De Persis and Tesi (2019), which applies behavioral systems theory to parametrize systems from past trajectories.…”

Section: Introductionmentioning

confidence: 99%

Data-driven Distributed and Localized Model Predictive Control

Alonso¹,

Yang²,

Matni³

2021

Preprint

View full text Add to dashboard Cite

Motivated by large-scale but computationally constrained settings, e.g., the Internet of Things, we present a novel data-driven distributed control algorithm that is synthesized directly from trajectory data. Our method, data-driven Distributed and Localized Model Predictive Control (D 3 LMPC), builds upon the data-driven System Level Synthesis (SLS) framework, which allows one to parameterize closed-loop system responses directly from collected open-loop trajectories. The resulting model-predictive controller can be implemented with distributed computation and only local information sharing. By imposing locality constraints on the system response, we show that the amount of data needed for our synthesis problem is independent of the size of the global system. Moreover, we show that our algorithm enjoys theoretical guarantees for recursive feasibility and asymptotic stability. Finally, we also demonstrate the optimality and scalability of our algorithm in a simulation experiment. * Denotes equal contribution.

show abstract

“…This inspires an increasing interest in understanding the performance of policy-based RL algorithms on simplified linear control benchmarks. For standard linear quadratic control problems, policy-based RL methods have been proved to yield strong convergence guarantees in various settings [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21]. For robust/risk-sensitive control problems, the robust adversarial reinforcement learning (RARL) framework appears to be quite relevant.…”

Section: Introductionmentioning

confidence: 99%

Model-Free $μ$ Synthesis via Adversarial Reinforcement Learning

Keivan¹,

Havens²,

Seiler³

et al. 2021

Preprint

View full text Add to dashboard Cite

Motivated by the recent empirical success of policy-based reinforcement learning (RL), there has been a research trend studying the performance of policy-based RL methods on standard control benchmark problems. In this paper, we examine the effectiveness of policy-based RL methods on an important robust control problem, namely µ synthesis. We build a connection between robust adversarial RL and µ synthesis, and develop a model-free version of the wellknown DK-iteration for solving state-feedback µ synthesis with static D-scaling. In the proposed algorithm, the K step mimics the classical central path algorithm via incorporating a recently-developed double-loop adversarial RL method as a subroutine, and the D step is based on model-free finite difference approximation. Extensive numerical study is also presented to demonstrate the utility of our proposed modelfree algorithm. Our study sheds new light on the connections between adversarial RL and robust control.

show abstract

On the Linear Convergence of Random Search for Discrete-Time LQR

Cited by 41 publications

References 7 publications

Learning Stabilizing Controllers of Linear Systems via Discount Policy Gradient

Learning Stabilizing Controllers of Linear Systems via Discount Policy Gradient

Data-driven Distributed and Localized Model Predictive Control

Model-Free $μ$ Synthesis via Adversarial Reinforcement Learning

Contact Info

Product

Resources

About