Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon

Abstract

We explore reinforcement learning methods for finding the optimal policy in the linear quadratic regulator (LQR) problem. In particular we consider the convergence of policy gradient methods in the setting of known and unknown parameters. We are able to produce a global linear convergence guarantee for this approach in the setting of finite time horizon and stochastic state dynamics under weak assumptions. The convergence of a projected policy gradient method is also established in order to handle problems with constraints. We illustrate the performance of the algorithm with two examples. The first example is the optimal liquidation of a holding in an asset. We show results for the case where we assume a model for the underlying dynamics and where we apply the method to the data directly. The empirical evidence suggests that the policy gradient method can learn the global optimal solution for a larger class of stochastic systems containing the LQR framework, and that it is more robust with respect to model misspecification when compared to a model-based approach. The second example is an LQR system in a higher dimensional setting with synthetic data.

Online Policy Gradient for Model Free Learning of Linear Quadratic Regulators with $\sqrt{T}$ Regret

“…Subsequently, Malik et al [20] improve the sample efficiency but their result holds only with a fixed probability and thus does not seem applicable for our purposes. Hambly et al [13] also improve the sample efficiency, but in a finite horizon setting. Mohammadi et al [22] give sample complexity bounds for the continuous-time variant of LQR.…”

Linear convergence of a policy gradient method for finite horizon continuous time stochastic control problems

“…Despite the notable success of PGMs, a mathematical theory that guarantees the convergence of these algorithms for general (continuous time) stochastic control problems has been elusive. Analysing the convergence behavior of PGMs is technically challenging, as the objective of a control problem is typically nonconvex with respect to the policies, even in the LQ setting [10,13]. Most existing theoretical results of PGMs, especially those establishing (optimal) linear convergence, focus on discrete time problems and restrict policies within specific parametric families.…”

“…Most existing theoretical results of PGMs, especially those establishing (optimal) linear convergence, focus on discrete time problems and restrict policies within specific parametric families. This includes Markov decision problems (MDPs) with softmax parameterized policies [26] or overparametrized one-hidden-layer neural-network policies [38,11,19], and discrete time linear (LQ) control problems with linear parameterized policies [10,13]. The analysis therein exploits heavily the specific structure of the considered (discrete time) control problems and policy parameterizations, and hence is difficult to extend to general continuous time control problems or general policy parameterizations.…”

“…The arguments therein often require specific problem structure, in order to derive a suitable Polyak-Lojasiewicz inequality (also known as the gradient dominance property) for the loss landscape. For instance, in the tabular MDP setting, the policies must be uniformly lower bounded away from zero over the entire state space [26], while in the LQ setting, eigenvalues of state covariance matrices must be lower bounded away from zero over the entire time horizon [10,13]. Consequently, these analyses are difficult to extend to general control problems (such as those with deterministic initial condition and degenerate noise) or to more sophisticated policy parameterizations (such as deep neural networks).…”

A Small Gain Analysis of Single Timescale Actor Critic

“…For policy gradient methods, a sample complexity of O(ǫ −2 ) is known; see e.g., [30] for a modern analysis, and see also [28] for speedup. If more structure about the problem is known and one considers an exact version, one can sometimes obtain a geometric rate; for example, [11] gives a geometric rate for a version of policy gradient when applied to an LQR problem. Likewise, information about about the structure of the optimal policy in general was shown to be useful in [21].…”