Adaptive Control and Regret Minimization in Linear Quadratic Gaussian (LQG) Setting

Lale, Sahin; Azizzadenesheli, Kamyar; Hassibi, Babak; Anandkumar, Anima

doi:10.23919/acc50511.2021.9483309

Cited by 10 publications

(4 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A series of papers (e.g. [1,4,5,20]) consider a model where a linear system with unknown dynamics is perturbed by stochastic disturbances; an online learner picks control actions with the goal of minimizing regret against the optimal stabilizing controller. In the "non-stochastic control" setting proposed in [15], the learner knows the dynamics, but the disturbance may be generated adversarially; the controller seeks to minimize regret against the class of disturbance-action policies.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Regret-Optimal Estimation and Control

Goel

Hassibi

2023

IEEE Trans. Automat. Contr.

View full text Add to dashboard Cite

We consider estimation and control in linear time-varying dynamical systems from the perspective of regret minimization. Unlike most prior work in this area, we focus on the problem of designing causal estimators and controllers which compete against a clairvoyant noncausal policy, instead of the best policy selected in hindsight from some fixed parametric class. We show that the regret-optimal estimator and regret-optimal controller can be derived in state-space form using operator-theoretic techniques from robust control and present tight, data-dependent bounds on the regret incurred by our algorithms in terms of the energy of the disturbances. Our results can be viewed as extending traditional robust estimation and control, which focuses on minimizing worst-case cost, to minimizing worst-case regret. We propose regret-optimal analogs of Model-Predictive Control (MPC) and the Extended Kalman Filter (EKF) for systems with nonlinear dynamics and present numerical experiments which show that our regret-optimal algorithms can significantly outperform standard approaches to estimation and control.

show abstract

Section: Related Workmentioning

confidence: 99%

“…where we initialize PT −1 = 0. The matrices Ãt , Bt , Ct , Lt , Kt , and Σt are defined in (19) and (20). The regret-optimal filter is the regret-suboptimal filter at level γ opt , where γ opt is the smallest value of γ such that Σt has the same inertia as the matrix I 0 0 −I .…”

Section: Regret-optimal Estimationmentioning

confidence: 99%

Regret-Optimal Estimation and Control

Goel

Hassibi

2023

IEEE Trans. Automat. Contr.

View full text Add to dashboard Cite

show abstract

“…Remark 2. We note that some works on learning based control have made assumptions that the underlying system is stable [28], or can be stabilized for all possible controls [15], when carrying out a regret analysis. This stands in contrast to Theorem 4 which merely requires that at least one of the M channels is stabilizing for the remote estimator, while some of the sub-optimal channels can cause the expected error covariance to diverge if they are used too often.…”

Section: Regret(t ) = θ(T )mentioning

confidence: 99%

Stability Enforced Bandit Algorithms for Channel Selection in State Estimation and Control

Leong¹,

Quevedo²,

Liu³

2022

Preprint

View full text Add to dashboard Cite

In this paper we consider a remote state estimation problem where a sensor can, at each discrete time instant, transmit on one out of M different communication channels. A key difficulty of the situation at hand is that the channel statistics are unknown. We study the case where both learning of the channel reception probabilities and state estimation is carried out simultaneously. Methods for choosing the channels based on techniques for multi-armed bandits are presented, and shown to provide stability of the remote estimator. Furthermore, we define the performance notion of estimation regret, and derive bounds on how it scales with time for the considered algorithms.

show abstract

“…Recent work [9] shows guarantees for related models in which the transition and observation dynamics are modeled by linear mixture models; however, their approach is computationally inefficient. We remark there are further works tackling online RL in other POMDPs, such as LQG [33,46], latent POMDPs [32] and reactive POMDPs [31,26].…”

Section: Related Workmentioning

confidence: 99%

Computationally Efficient PAC RL in POMDPs with Latent Determinism and Conditional Embeddings

Uehara¹,

Sekhari²,

Lee³

et al. 2022

Preprint

View full text Add to dashboard Cite

We study reinforcement learning with function approximation for large-scale Partially Observable Markov Decision Processes (POMDPs) where the state space and observation space are large or even continuous. Particularly, we consider Hilbert space embeddings of POMDP where the feature of latent states and the feature of observations admit a conditional Hilbert space embedding of the observation emission process, and the latent state transition is deterministic. Under the function approximation setup where the optimal latent stateaction Q-function is linear in the state feature, and the optimal Q-function has a gap in actions, we provide a computationally and statistically efficient algorithm for finding the exact optimal policy. We show our algorithm's computational and statistical complexities scale polynomially with respect to the horizon and the intrinsic dimension of the feature on the observation space. Furthermore, we show both the deterministic latent transitions and gap assumptions are necessary to avoid statistical complexity exponential in horizon or dimension. Since our guarantee does not have an explicit dependence on the size of the state and observation spaces, our algorithm provably scales to large-scale POMDPs.

show abstract

Adaptive Control and Regret Minimization in Linear Quadratic Gaussian (LQG) Setting

Cited by 10 publications

References 18 publications

Regret-Optimal Estimation and Control

Regret-Optimal Estimation and Control

Stability Enforced Bandit Algorithms for Channel Selection in State Estimation and Control

Computationally Efficient PAC RL in POMDPs with Latent Determinism and Conditional Embeddings

Contact Info

Product

Resources

About