On a Phase Transition of Regret in Linear Quadratic Control: The Memoryless Case

Ziemann, Ingvar

doi:10.1109/lcsys.2020.3005224

Cited by 2 publications

(1 citation statement)

References 18 publications

(40 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Notably, [SF20] provides nearly matching upper and lower bounds scaling almost correctly with the dimensional dependence given that the entire set of parameters (A, B) are unknown. See also [CCK20,ZS20a]. Moreover, we wish to mention that the present work is an extension of an earlier conference paper [ZS20b], which gives regret lower bounds for the particular case when the matrix B is unknown.…”

Section: Related Workmentioning

confidence: 85%

Regret Lower Bounds for Learning Linear Quadratic Gaussian Systems

Ziemann¹

2022

Preprint

Self Cite

View full text Add to dashboard Cite

This paper presents local minimax regret lower bounds for adaptively controlling linear-quadratic-Gaussian (LQG) systems. We consider smoothly parametrized instances and provide an understanding of when logarithmic regret is impossible which is both instance specific and flexible enough to take problem structure into account. This understanding relies on two key notions: That of local-uninformativeness; when the optimal policy does not provide sufficient excitation for identification of the optimal policy, and yields a degenerate Fisher information matrix; and that of information-regretboundedness, when the small eigenvalues of a policy-dependent information matrix are boundable in terms of the regret of that policy. Combined with a reduction to Bayesian estimation and application of Van Trees' inequality, these two conditions are sufficient for proving regret bounds on order of magnitude √ T in the time horizon, T . This method yields lower bounds that exhibit tight dimensional dependencies and scale naturally with control-theoretic problem constants. For instance, we are able to prove that systems operating near marginal stability are fundamentally hard to learn to control. We further show that large classes of systems satisfy these conditions, among them any state-feedback system with both A-and B-matrices unknown. Most importantly, we also establish that a nontrivial class of partially observable systems, essentially those that are over-actuated, satisfy these conditions, thus providing a √ T lower bound also valid for partially observable systems. Finally, we turn to two simple examples which demonstrate that our lower bound captures classical control-theoretic intuition: our lower bounds diverge for systems operating near marginal stability or with large filter gain -these can be arbitrarily hard to (learn to) control.

show abstract

Section: Related Workmentioning

confidence: 85%