A Curse-of-Dimensionality-Free Numerical Method for Solution of Certain HJB PDEs

Abstract

In previous work of the author and others, max-plus methods have been explored for solution of first-order, nonlinear Hamilton-Jacobi-Bellman partial differential equations (HJB PDEs) and corresponding nonlinear control problems. These methods exploit the max-plus linearity of the associated semigroups. In particular, although the problems are nonlinear, the semigroups are linear in the max-plus sense. These methods have been used successfully to compute solutions. Although they provide certain computational-speed advantages, they still generally suffer from the curse-of-dimensionality. Here we consider HJB PDEs where the Hamiltonian takes the form of a (pointwise) maximum of linear/quadratic forms. The approach to solution will be rather general, but in order to ground the work, we consider only constituent Hamiltonians corresponding to long-run average-cost-per-unit-time optimal control problems for the development. We obtain a numerical method not subject to the curse-of-dimensionality. The method is based on construction of the dual-space semigroup corresponding to the HJB PDE. This dual-space semigroup is constructed from the dual-space semigroups corresponding to the constituent linear/quadratic Hamiltonians. The dual-space semigroup is particularly useful due to its form as a max-plus integral operator with kernel obtained from the originating semigroup. One considers repeated application of the dual-space semigroup to obtain the solution.

Distributed dynamic programming for discrete-time stochastic control, and idempotent algorithms

“…Theorem 4.2 Suppose V t is given by (16), where g t is Borel measurable and satisfies (19), and (Z t , d zt ) is a separable metric space. Also assume that given R < ∞, there exists D t,R < ∞ such that…”

“…This will allow us to briefly discuss the curse-of-complexity, as it will appear in such stochastic control problems. (Note that although these methods eliminate the curse-ofdimensionality, they are subject to curse-of-complexity difficulties [11], [16], [17]; methods for attenuation of the curse-of-complexity are discussed in [12], [13], [21].…”

“…As an aside, it is useful to note that quadratic Hamiltonians form a min-plus basis for the space of semiconvcave Hamiltonians, and that this (in the max-plus/semiconvex case) was used for deterministic control problems [11], [13], [14], [16], [17]. Further, we will see that in such cases, the g t will all be quadratics, and so backward propagation via the IDDPP reduces to propagation of the coefficients in the quadratic forms.…”

“…Using this technology, the value function will be propagated backwards with a representation as a pointwise minimum of quadratic or affine forms. This approach will have some similarities to the idempotent curse-ofdimensionality-free methods developed for deterministic control problems [11], [13], [14], [16]. However, the need to use the idempotent distributive property makes this approach very different from those.…”

“…Here, max-plus methods are appropriate for problems with maximizing controllers and vice-versa. These methods include maxplus basis-expansion approaches [1], [2], [6], [7], [11], [15], [18], as well as the more recently developed curse-of-dimensionality-free methods [11], [16], [17]. However, stochastic control problems have eluded idempotent methods.…”