A moment and sum-of-squares extension of dual dynamic programming with application to nonlinear energy storage problems

Abstract

We present a finite-horizon optimization algorithm that extends the established concept of Dual Dynamic Programming (DDP) in two ways. First, in contrast to the linear costs, dynamics, and constraints of standard DDP, we consider problems in which all of these can be polynomial functions. Second, we allow the state trajectory to be described by probability distributions rather than point values, and return approximate value functions fitted to these. The algorithm is in part an adaptation of sum-of-squares techniques used in the approximate dynamic programming literature. It alternates between a forward simulation through the horizon, in which the moments of the state distribution are propagated through a succession of single-stage problems, and a backward recursion, in which a new polynomial function is derived for each stage using the moments of the state as fixed data. The value function approximation returned for a given stage is the pointwise maximum of all polynomials derived for that stage. This contrasts with the piecewise affine functions derived in conventional DDP. We prove key convergence properties of the new algorithm, and validate it in simulation on two case studies related to the optimal operation of energy storage devices with nonlinear characteristics. The first is a small borehole storage problem, for which multiple value function approximations can be compared. The second is a larger problem, for which conventional discretized dynamic programming is intractable.

Two-Stage Dual Dynamic Programming With Application to Nonlinear Hydro Scheduling

“…The second inequality (15) comes from substituting in the right inequality of (13) into (14). The final inequality (16) is…”

“…Since DDP cannot solve problems with nonconvex value functions, convex approximations of the model are required when using DDP for nonlinear problems. Recent extensions of this approach have considered integer programs [12], locally-valid Benders cuts [13], and polynomial-based moment relaxation [14].…”

Learning continuous $Q$-functions using generalized Benders cuts

“…Examples include the quadratic lower bound in [14], and the polynomial derived using sum-of-squares techniques in [13]. Approximate Vfunctions represented as the pointwise maximum of multiple lower-bounding functions have been used in [1], [9], [10], [15]. Recent work utilizing a point-wise maximum representation [16] has extended the Benders decomposition argument used for linear multi-stage decision problems in Dual DP (DDP, [12]), to a general nonlinear, infinite-horizon setting.…”

“…Thus the strong duality condition in Assumption 3 holds.To prove Lipschitz continuity in Assumption 3, one must bound the gradient in u-space of the functions q i (x m , ·) for any given x m . Inspection of problem(9) shows that each new function q I+1 depends on the existing functions q 0 , . .…”

Nonlinear Control of Quadcopters via Approximate Dynamic Programming

“…This reduces the dependency of the result on the choice of the individual objectives. Parts of these techniques are used in [13] and demonstrated in simulation on a nonlinear energy storage system with a low dimensional state-by-input space. In that application, the slow time scale allows for computationally demanding online policies.…”