Explore More and Improve Regret in Linear Quadratic Regulators

Lale, Sahin; Azizzadenesheli, Kamyar; Hassibi, Babak; Anandkumar, Anima

doi:10.48550/arxiv.2007.12291

Cited by 8 publications

(23 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…i . Unlike related works on system identification and regret analysis [15,36,37,52,59], mean-square stability does not lead to strong high-probability bounds, as one can only bound x t or x t x ⊺ t in expectation. Therefore, in Algorithm 1, we sample only bounded state-excitation pairs (x t , z t ) on each mode i ∈ [s].…”

Section: System Identification For Mjsmentioning

confidence: 99%

“…A commonly used control paradigm is the Linear Quadratic Regulator (LQR), which is theoretically well understood when system dynamics are linear and known. LQR also provides an interesting benchmark, when system dynamics are unknown, for reinforcement learning (RL) with continuous state and action spaces and for adaptive control [2,4,9,16,36,47].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Identification and Adaptive Control of Markov Jump Systems: Sample Complexity and Regret Bounds

Sattar¹,

Du²,

Tarzanagh³

et al. 2021

Preprint

View full text Add to dashboard Cite

Learning how to effectively control unknown dynamical systems is crucial for intelligent autonomous systems. This task becomes a significant challenge when the underlying dynamics are changing with time. Motivated by this challenge, this paper considers the problem of controlling an unknown Markov jump linear system (MJS) to optimize a quadratic objective. By taking a model-based perspective, we consider identification-based adaptive control for MJSs. We first provide a system identification algorithm for MJS to learn the dynamics in each mode as well as the Markov transition matrix, underlying the evolution of the mode switches, from a single trajectory of the system states, inputs, and modes. Through mixing-time arguments, sample complexity of this algorithm is shown to be O(1 √ T ). We then propose an adaptive control scheme that performs system identification together with certainty equivalent control to adapt the controllers in an episodic fashion. Combining our sample complexity results with recent perturbation results for certainty equivalent control, we prove that when the episode lengths are appropriately chosen, the proposed adaptive control scheme achieves O( √ T ) regret, which can be improved to O(polylog(T )) with partial knowledge of the system. Our proof strategy introduces innovations to handle Markovian jumps and a weaker notion of stability common in MJSs. Our analysis provides insights into system theoretic quantities that affect learning accuracy and control performance. Numerical simulations are presented to further reinforce these insights.

show abstract

Section: System Identification For Mjsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Identification and Adaptive Control of Markov Jump Systems: Sample Complexity and Regret Bounds

Sattar¹,

Du²,

Tarzanagh³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Most existing work assume that we have access to a stabilizer. Recent attempts to remove this assumption include [LAHA20]. There, for example, the authors assume that the algorithm knows that the system (A, B) belongs to a set of systems…”

Section: A Related Workmentioning

confidence: 99%

“…In [LAHA20], the authors provide another improvement on the OFU based algorithm of [AYS11]. Their goal is to improve the dependency of the regret upper bound on the dimension and to remove the assumption of having access to a stabilizing controller.…”

Section: Ofu-based Algorithmsmentioning

confidence: 99%

Minimal Expected Regret in Linear Quadratic Control

Jedra¹,

Proutière²

2021

Preprint

View full text Add to dashboard Cite

We consider the problem of online learning in Linear Quadratic Control systems whose state transition and state-action transition matrices A and B may be initially unknown. We devise an online learning algorithm and provide guarantees on its expected regret. This regret at time T is upper bounded (i) by O((du + dx) √ dxT ) when A and B are unknown, (ii) by O(d 2x log(T )) if only A is unknown, and (iii) by O(dx(du + dx) log(T )) if only B is unknown and under some mild non-degeneracy condition (dx and du denote the dimensions of the state and of the control input, respectively). These regret scalings are minimal in T , dx and du as they match existing lower bounds in scenario (i) when dx ≤ du [SF20], and in scenario (ii) [Lai86]. We conjecture that our upper bounds are also optimal in scenario (iii) (there is no known lower bound in this setting).Existing online algorithms proceed in epochs of (typically exponentially) growing durations. The control policy is fixed within each epoch, which considerably simplifies the analysis of the estimation error on A and B and hence of the regret. Our algorithm departs from this design choice: it is a simple variant of certainty-equivalence regulators, where the estimates of A and B and the resulting control policy can be updated as frequently as we wish, possibly at every step. Quantifying the impact of such a constantly-varying control policy on the performance of these estimates and on the regret constitutes one of the technical challenges tackled in this paper.

show abstract

“…(3) Online LQR with Unknown Dynamics and Time-Invariant Costs: There is a recent line of research dealing with LQR control problems with unknown dynamics. Several techniques are proposed using (i) gradient estimation (see e.g., [31]- [34]) (ii) the estimation of dynamics matrices and derivation of the controller by considering the estimation uncertainty ( [7], [8], [23], [35]- [37]), and (iii) wave-filtering [38], [39].…”

Section: A Related Workmentioning

confidence: 99%

Regret Analysis of Distributed Online LQR Control for Unknown LTI Systems

Chang

Shahrampour

2021

Preprint

View full text Add to dashboard Cite

Online learning has recently opened avenues for rethinking classical optimal control beyond time-invariant cost metrics, and online controllers are designed when the performance criteria changes adversarially over time. Inspired by this line of research, we study the distributed online linear quadratic regulator (LQR) problem for linear time-invariant (LTI) systems with unknown dynamics. Consider a multi-agent network where each agent is modeled as a LTI system. The LTI systems are associated with time-varying quadratic costs that are revealed sequentially. The goal of the network is to collectively (i) estimate the unknown dynamics and (ii) compute local control sequences competitive to that of the best centralized policy in hindsight that minimizes the sum of costs for all agents. This problem is formulated as a regret minimization. We propose a distributed variant of the online LQR algorithm where each agent computes its system estimate during an exploration stage. The agent then applies distributed online gradient descent on a semi-definite programming (SDP) whose feasible set is based on the agent's system estimate. We prove that the regret bound of our proposed algorithm scales Õ(T 2/3 ), implying the consensus of the network over time. We also provide simulation results verifying our theoretical guarantee.

show abstract

Explore More and Improve Regret in Linear Quadratic Regulators

Cited by 8 publications

References 2 publications

Identification and Adaptive Control of Markov Jump Systems: Sample Complexity and Regret Bounds

Identification and Adaptive Control of Markov Jump Systems: Sample Complexity and Regret Bounds

Minimal Expected Regret in Linear Quadratic Control

Regret Analysis of Distributed Online LQR Control for Unknown LTI Systems

Contact Info

Product

Resources

About