Approximating optimal feedback controllers of finite horizon control problems using hierarchical tensor formats

Abstract

Controlling systems of ordinary differential equations (ODEs) is ubiquitous in science and engineering. For finding an optimal feedback controller, the value function and associated fundamental equations such as the Bellman equation and the Hamilton-Jacobi-Bellman (HJB) equa-

Dynamical low-rank approximations of solutions to the Hamilton-Jacobi-Bellman equation

“…For recent applications of TTs as value function approximators, see e.g. [KKD19;OSS21a]. Solution methods based on high-dimensional polynomials and tensor spaces have also been considered in [DKK21;KK18].…”

“…This approach is based on Bellman's principle. However, in contrast to comparable recent work [OSS21a], we use the HJB equation (4) on each subinterval instead of the Bellman equation. In particular, we define suitable approximate solutions to the HJB equation by means of the Dirac-Frenkel variational principle.…”

“…TT approximations of the value function by means of such a backwards scheme were already presented e.g. in [OSS21a]. In that work however, the integral formulation (6) is used exclusively, sampling trajectories x(t) for given controls and adding up the costs.…”

“…As a benchmark for assessing the performance of our method, we use the TT-based approach from [OSS21a] with the same hyper-parameters. To make this precise, instead of solving (24) by means of our dynamical low-rank scheme, Vi is approximated in each policy iteration step by sampling the trajectories x k (t), t ∈ [t i , t i+1 ] of all sample points.…”

“…Tree based tensor networks and tensor trains in particular have already been used for successful approximations of the value function in various works, see e.g. [Fac+20;KKD19;OSS21a]. These recent results are summarized in the PhD thesis of Leon Sallandt [Sal21], which is still being finalised as this paper is written.…”