Implicit Differential Dynamic Programming

Abstract

Over the past decade, the Differential Dynamic Programming (DDP) method has gained in maturity and popularity within the robotics community. Several recent contributions have led to the integration of constraints within the original DDP formulation, hence enlarging its domain of application while making it a strong and easy-to-implement competitor against alternative methods of the state of the art such as collocation or multiple-shooting approaches. Yet, and similarly to its competitors, DDP remains unable to cope with high-dimensional dynamics within a receding horizon fashion, such as in the case of online generation of athletic motion on humanoid robots. In this paper, we propose to make a step toward this objective by reformulating classic DDP as an implicit optimal control problem, allowing the use of more advanced integration schemes such as implicit or variational integrators. To that end, we introduce a primal-dual proximal Lagrangian approach capable of handling dynamic and path constraints in a unified manner, while taking advantage of the time sparsity inherent to optimal control problems. We show that his reformulation enables us to relax the dynamics along the optimization process by solving it inexactly: far from the optimality conditions, the dynamics are only partially fulfilled, but continuously enforced as the solver get closer to the local optimal solution. This inexactness enables our approach to robustly handle large time steps (100 ms or more), unlike other DDP solvers of the state of the art, as experimentally validated through different robotic scenarios.

Constrained Differential Dynamic Programming: A primal-dual augmented Lagrangian approach

“…In [20], we show that applying the MM to (2) leads to a relaxation of the Bellman equation, in the equalityconstrained case. We now extend this idea to the inequalityconstrained case.…”

“…Yet, we choose to maintain the ν variable relaxed in the minimization procedure (20) as it enables us to preserve similar well conditioned linear systems and stopping criterion derived in (12) and (14). Consequently, the generalized merit function corresponds to:…”

“…In this section, we extend the differential dynamic programming framework accounting for equality constraints [18] and implicit dynamics [20] to the case of inequality constraints using the PDAL introduced in Sec. II.…”

“…Their method converges reliably to good numerical accuracy. We have recently extended this formulation to also account for the dynamics and other equality constraints using a primal-dual augmented Lagrangian, allowing for the inclusion of infeasible initialization and implicit integrators [20].…”

Enforcing the consensus between Trajectory Optimization and Policy Learning for precise robot control

“…Such an approach is made possible by the recent emergence of differentiable physics engines, relying on derivatives of rigid body algorithms [12] and frictional contacts [13], [14]. A second key ingredient for model-based approaches is access to an efficient numerical solvers for trajectory optimization [4], [15]. Indeed, specialized OC algorithms exploit physical models, their derivatives, and the inherent temporal structure of the problem via the Bellman equation (or Pontryagin's principle).…”