Newton’s Method and Differential Dynamic Programming for Unconstrained Nonlinear Dynamic Games

Abstract

Dynamic games arise when multiple agents with differing objectives choose control a dynamic system. They model a wide variety of applications in economics, defense, energy systems and etc. However, compared to single-agent control problems, the computational methods for dynamic games are relatively limited. As in the single-agent case, only specific dynamic games can be solved exactly, and so approximation algorithms are required. In this paper, we show how to extend a recursive Newton's algorithm and the popular differential dynamic programming (DDP) for single-agent optimal control to the case of full-information non-zero sum dynamic games. In the singleagent case, convergence of DDP is proved by comparison with Newton's method, which converges locally at a quadratic rate. We show that the iterates of Newton's method and DDP are sufficiently close for the DDP to inherit the quadratic convergence rate of Newton's method. We also prove both methods result in an open-loop Nash equilibrium and a local feedback O(ε 2 )-Nash equilibrium. Numerical examples are provided.

Inferring Objectives in Continuous Dynamic Games from Noise-Corrupted Partial State Observations

“…Recent algorithms for open-loop games are those by Di and Lamperski [8], Le Cleac'h et al [19], and for feedback games we refer to Fridovich-Keil et al [11] and Laine et al [18].…”

“…the solution to a joint optimal control problem. Despite the added complexity of these noncooperative interactions, recent developments enable computationally-efficient solutions to the dynamic games which arise in multi-agent robotic settings [8,11,12,19]. There are similar challenges in designing objectives for dynamic games as for single-player optimal control problems.…”

ALGAMES: A Fast Solver for Constrained Dynamic Games

“…Nash equilibria have been investigated in [4,5,8,14,15,16,17]. We also take the approach of searching for Nash Fig.…”

“…Third, algorithms akin to differential dynamic programming have been developed for robust control [19] and later applied to game-theoretic problems [4,15]. This approach scales polynomially with the number of players and is fast enough to run real-time in a model-predictive control (MPC) fashion [4].…”

“…This behavior can be overly aggressive for the leader (or overly passive for the follower) and does not capture the game-theoretic nature of the problem. Nash equilibria have been investigated in [4], [5], [8], [13], [14], [15], [16]. We also take the approach of searching for Nash equilibria, as this type of equilibrium seems better suited to symmetric, multi-robot interaction scenarios.…”