Learning Linear-Quadratic Regulators Efficiently with only $\sqrt{T}$ Regret

Cohen, Alon; Koren, Tomer; Mansour, Yishay

doi:10.48550/arxiv.1902.06223

Cited by 26 publications

(80 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We remark that despite being linear, the Markov transition model P h (•|x, a) can still have infinite degrees of freedom as the measure µ h is unknown. This is a key difference from the linear quadratic regulator [1,18,4,3,15] or the recent work of Yang and Wang [50], whose transition models are completely specified by a finite-dimensional matrix such that the degrees of freedom are bounded.…”

Section: Linear Markov Decision Processesmentioning

confidence: 99%

Provably Efficient Reinforcement Learning with Linear Function Approximation

Jin

Yang

Wang

et al. 2019

Preprint

202

View full text Add to dashboard Cite

Modern Reinforcement Learning (RL) is commonly applied to practical problems with an enormous number of states, where function approximation must be deployed to approximate either the value function or the policy. The introduction of function approximation raises a fundamental set of challenges involving computational and statistical efficiency, especially given the need to manage the exploration/exploitation tradeoff. As a result, a core RL question remains open: how can we design provably efficient RL algorithms that incorporate function approximation? This question persists even in a basic setting with linear dynamics and linear rewards, for which only linear function approximation is needed.This paper presents the first provable RL algorithm with both polynomial runtime and polynomial sample complexity in this linear setting, without requiring a "simulator" or additional assumptions. Concretely, we prove that an optimistic modification of Least-Squares Value Iteration (LSVI)-a classical algorithm frequently studied in the linear setting-achieves O( √ d 3 H 3 T ) regret, where d is the ambient dimension of feature space, H is the length of each episode, and T is the total number of steps. Importantly, such regret is independent of the number of states and actions.

show abstract

Section: Linear Markov Decision Processesmentioning

confidence: 99%

Provably Efficient Reinforcement Learning with Linear Function Approximation

Jin

Yang

Wang

et al. 2019

Preprint

202

View full text Add to dashboard Cite

show abstract

“…Faradonbeh et al [15] argue that certainty equivalence control with an epsilon-greedy-like scheme achieves O( √ T ) regret, though their work does not provide any explicit dependencies on instance parameters. Finally, Cohen et al [9] also give an efficient algorithm based on semidefinite programming that achieves O( √ T ) regret. The literature for LQG is less complete, with most of the focus on the estimation side.…”

Section: Related Workmentioning

confidence: 99%

Certainty Equivalence is Efficient for Linear Quadratic Control

Mania,

Tu,

Recht

2019

Preprint

View full text Add to dashboard Cite

We study the performance of the certainty equivalent controller on Linear Quadratic (LQ) control problems with unknown transition dynamics. We show that for both the fully and partially observed settings, the sub-optimality gap between the cost incurred by playing the certainty equivalent controller on the true system and the cost incurred by using the optimal LQ controller enjoys a fast statistical rate, scaling as the square of the parameter error. To the best of our knowledge, our result is the first sub-optimality guarantee in the partially observed Linear Quadratic Gaussian (LQG) setting. Furthermore, in the fully observed Linear Quadratic Regulator (LQR), our result improves upon recent work by Dean et al. [10], who present an algorithm achieving a sub-optimality gap linear in the parameter error. A key part of our analysis relies on perturbation bounds for discrete Riccati equations. We provide two new perturbation bounds, one that expands on an existing result from Konstantinov et al. [24], and another based on a new elementary proof strategy.

show abstract

“…A series of papers (e.g. [1,4,5,20]) consider a model where a linear system with unknown dynamics is perturbed by stochastic disturbances; an online learner picks control actions with the goal of minimizing regret against the optimal stabilizing controller. In the "non-stochastic control" setting proposed in [15], the learner knows the dynamics, but the disturbance may be generated adversarially; the controller seeks to minimize regret against the class of disturbance-action policies.…”

Section: Related Workmentioning

confidence: 99%

Regret-optimal Estimation and Control

Goel¹,

Hassibi²

2021

Preprint

View full text Add to dashboard Cite

We consider estimation and control in linear time-varying dynamical systems from the perspective of regret minimization. Unlike most prior work in this area, we focus on the problem of designing causal estimators and controllers which compete against a clairvoyant noncausal policy, instead of the best policy selected in hindsight from some fixed parametric class. We show that the regret-optimal estimator and regret-optimal controller can be derived in state-space form using operator-theoretic techniques from robust control and present tight, data-dependent bounds on the regret incurred by our algorithms in terms of the energy of the disturbances. Our results can be viewed as extending traditional robust estimation and control, which focuses on minimizing worst-case cost, to minimizing worst-case regret. We propose regret-optimal analogs of Model-Predictive Control (MPC) and the Extended Kalman Filter (EKF) for systems with nonlinear dynamics and present numerical experiments which show that our regret-optimal algorithms can significantly outperform standard approaches to estimation and control.

show abstract

Learning Linear-Quadratic Regulators Efficiently with only $\sqrt{T}$ Regret

Abstract: We present the first computationally-efficient algorithm with O( √ T ) regret for learning in Linear Quadratic Control systems with unknown dynamics. By that, we resolve an open question of Tu (2018).

Cited by 26 publications

References 4 publications

Provably Efficient Reinforcement Learning with Linear Function Approximation

Provably Efficient Reinforcement Learning with Linear Function Approximation

Certainty Equivalence is Efficient for Linear Quadratic Control

Regret-optimal Estimation and Control

Contact Info

Product

Resources

About