Regret-Optimal Controller for the Full-Information Problem

Abstract

We consider the infinite-horizon, discrete-time fullinformation control problem. Motivated by learning theory, as a criterion for controller design we focus on regret, defined as the difference between the LQR cost of a causal controller (that has only access to past and current disturbances) and the LQR cost of a clairvoyant one (that has also access to future disturbances). In the full-information setting, there is a unique optimal non-causal controller that in terms of LQR cost dominates all other controllers, and we focus on the regret compared to this particular controller. Since the regret itself is a function of the disturbances, we consider the worst-case regret over all possible bounded energy disturbances, and propose to find a causal controller that minimizes this worst-case regret. The resulting controller has the interpretation of guaranteeing the smallest possible regret compared to the best non-causal controller that can see the future, no matter what the disturbances are. We show that the regretoptimal control problem can be reduced to a Nehari extension problem, i.e., to approximate an anticausal operator with a causal one in the operator norm. In the state-space setting we obtain explicit formulas for the optimal regret and for the regret-optimal controller (in both the causal and the strictly causal settings). The regret-optimal controller is the sum of the classical H2 state-feedback law and an n-th order controller (where n is the state dimension of the plant) obtained from the Nehari problem. The controller construction simply requires the solution to the standard LQR Riccati equation, in addition to two Lyapunov equations. Simulations over a range of plants demonstrates that the regret-optimal controller interpolates nicely between the H2 and the H∞ optimal controllers, and generally has H2 and H∞ costs that are simultaneously close to their optimal values. The regret-optimal controller thus presents itself as a viable option for control system design.

Regret-Optimal Full-Information Control

“…The sets of the causal and strcitly causal policies are denoted by Π C and Π S.C. respectively. Moreover, we restrict our attention in both scenarios to linear policies 1 .…”

Optimal Competitive-Ratio Control

“…Motivated by competitive design approaches in online learning [3], two control paradigms have been recently proposed as alternatives to the classical paradigms, namely regret-optimal control [4] and competitive-ratio control [5]. These criteria aim to minimize the difference or the ratio between the costs of a causal controller (to be designed) and the cost of the optimal clairvoyant controller that has access to entire disturbances sequence.…”

“…In the competitive-ratio case, we minimize the corresponding worst-case ratio of these two costs. These strategies result in novel controller behavior, e.g., the regret-optimal controller interpolates between H 2 and H ∞ to achieve the best of both worlds [4], and provide superior performance in various control tasks with various disturbance characteristics [4][5][6].…”

“…The regret-optimal control problem has been solved in [4], meaning that the optimal value of the regret has been explicitly given and a corresponding state-space controller has been provided [7]. In the competitive ratio case, however, only the sub-optimal problem (where the competitive ratio is bounded by a given constant) has been solved, meaning that the optimal solution can only be found by performing a bisection over this constant [5] 1 .…”

“…This framework aims to produce controllers mimic the performance of the clairvoyant controller by minimizing the regret. The formulation of the regret-optimal control problem and the regret-optimal controller were introduced in [4,7] for the full-information control problem, and was later extended to the finite-horizon regime in [23]. The controller for the finite-horizon regime has been extended to incorporate state and input safety constraints in [24,25] by leveraging the System Level Synthesis framework [26], and to regret bound for the H ∞ controller [27].…”