Regret-Optimal Full-Information Control

Abstract

We consider the infinite-horizon, discrete-time fullinformation control problem. Motivated by learning theory, as a criterion for controller design we focus on regret, defined as the difference between the LQR cost of a causal controller (that has only access to past and current disturbances) and the LQR cost of a clairvoyant one (that has also access to future disturbances). In the full-information setting, there is a unique optimal non-causal controller that in terms of LQR cost dominates all other controllers, and we focus on the regret compared to this particular controller. Since the regret itself is a function of the disturbances, we consider the worst-case regret over all possible bounded energy disturbances, and propose to find a causal controller that minimizes this worst-case regret. The resulting controller has the interpretation of guaranteeing the smallest possible regret compared to the best non-causal controller that can see the future, no matter what the disturbances are. We show that the regretoptimal control problem can be reduced to a Nehari extension problem, i.e., to approximate an anticausal operator with a causal one in the operator norm. In the state-space setting we obtain explicit formulas for the optimal regret and for the regret-optimal controller (in both the causal and the strictly causal settings). The regret-optimal controller is the sum of the classical H2 state-feedback law and an n-th order controller (where n is the state dimension of the plant) obtained from the Nehari problem. The controller construction simply requires the solution to the standard LQR Riccati equation, in addition to two Lyapunov equations. Simulations over a range of plants demonstrates that the regret-optimal controller interpolates nicely between the H2 and the H∞ optimal controllers, and generally has H2 and H∞ costs that are simultaneously close to their optimal values. The regret-optimal controller thus presents itself as a viable option for control system design.

Competitive Control

“…Dynamic regret is a very similar metric to competitive ratio, which we consider in this paper, except that it is the difference between the cost of the online and offline controllers, rather than the ratio of the costs. The problem of designing controllers with optimal dynamic regret was studied in the finite-horizon, timevarying setting in [8], in the infinite-horizon LTI setting in [17], and in the measurement-feedback setting in [9]. Gradientbased algorithms with low dynamic regret against the class of disturbance-action policies were obtained in [12], [19].…”

“…A controller whose competitive ratio is bounded above by C offers the following guarantee: the cost it incurs is always at most a factor of C higher than the cost that could have been counterfactually incurred by any other controller, irrespective of the disturbance is generated. Competitive ratio is a multiplicative analog of dynamic regret; the problem of obtaining controllers with optimal dynamic regret was recently considered in [8], [9], [17].…”

Online estimation and control with optimal pathlength regret

“…The problem of designing controllers with optimal dynamic regret was studied in the finite-horizon, timevarying setting in [5], in the infinite-horizon LTI setting in [12], and in the measurement-feedback setting in [6]. These works all bounded regret by the energy in the disturbances; the pathlength regret bounds we obtain in this paper also imply energy regret bounds which are optimal up to a factor of 4.…”

“…These works all bounded regret by the energy in the disturbances; the pathlength regret bounds we obtain in this paper also imply energy regret bounds which are optimal up to a factor of 4. Filtering algorithms with energy regret bounds were obtained in the finite-horizon setting in [5] and the infinite-horizon setting in [12]. Gradient-based control algorithms with low dynamic regret against the class of disturbance-action policies were obtained in [15]; the stronger metric of adaptive regret was studied in [8].…”

“…Theorem 2 (Theorem 9 in [12]). The smallest value of γ such that there exists a causal and bounded K(z) satisfying ( 7) is γ = σ(ZΠ), where Z and Π are solutions of the Lyapunov equations…”