Certainty Equivalence is Efficient for Linear Quadratic Control

Abstract

We study the performance of the certainty equivalent controller on Linear Quadratic (LQ) control problems with unknown transition dynamics. We show that for both the fully and partially observed settings, the sub-optimality gap between the cost incurred by playing the certainty equivalent controller on the true system and the cost incurred by using the optimal LQ controller enjoys a fast statistical rate, scaling as the square of the parameter error. To the best of our knowledge, our result is the first sub-optimality guarantee in the partially observed Linear Quadratic Gaussian (LQG) setting. Furthermore, in the fully observed Linear Quadratic Regulator (LQR), our result improves upon recent work by Dean et al. [10], who present an algorithm achieving a sub-optimality gap linear in the parameter error. A key part of our analysis relies on perturbation bounds for discrete Riccati equations. We provide two new perturbation bounds, one that expands on an existing result from Konstantinov et al. [24], and another based on a new elementary proof strategy.

Online Actuator Selection and Controller Design for Linear Quadratic Regulation with Unknown System Model

“…This way of designing the control policy is also known as the certainty equivalence approach (e.g., [4]). Specifically, the authors in [10,25] provided an online algorithm for the LQR problem with unknown system matrices and showed that the regret of the algorithm is Õ( √ N ), where N is the number of time steps in the LQR problem and Õ(•) hides logarithmic factors in N . Note that the authors in [1,11,10,25] considered the infinite horizon LQR setting.…”

“…Specifically, the authors in [10,25] provided an online algorithm for the LQR problem with unknown system matrices and showed that the regret of the algorithm is Õ( √ N ), where N is the number of time steps in the LQR problem and Õ(•) hides logarithmic factors in N . Note that the authors in [1,11,10,25] considered the infinite horizon LQR setting. We extend the analyses and results in [25] to the finite horizon LQR setting when solving the problem considered in this paper.…”

“…Note that the authors in [1,11,10,25] considered the infinite horizon LQR setting. We extend the analyses and results in [25] to the finite horizon LQR setting when solving the problem considered in this paper.…”

“…We analyze the regret of the proposed algorithm and show that the regret is Õ( √ T ) with high probability, where T is the number of rounds in the problem and Õ(•) hides logarithmic factors in T . When analyzing the regret of our algorithm, we also extend the certainty equivalence approach proposed in [25] for learning LQR over an infinite horizon to the finite-horizon setting.…”

“…For the controller design, we leverage the certainty equivalence approach, which has been studied for the LQR problem over an infinite horizon with unknown system model (e.g., [11,25]). Specifically, in the certainty equivalence approach, we design a control policy based on estimated system matrices, denoted as  and B.…”