A Neural Network Approach for High-Dimensional Optimal Control Applied to Multi-Agent Path Finding

Abstract

We propose a neural network approach for solving high-dimensional optimal control problems arising in real-time applications. Our approach yields controls in a feedback form and can therefore handle uncertainties such as perturbations to the system's state. We accomplish this by fusing the Pontryagin Maximum Principle (PMP) and Hamilton-Jacobi-Bellman (HJB) approaches and parameterizing the value function with a neural network. We train our neural network model using the objective function of the control problem and penalty terms that enforce the HJB equations. Therefore, our training algorithm does not involve data generated by another algorithm. By training on a distribution of initial states, we ensure the controls' optimality on a large portion of the state-space. Our grid-free approach scales efficiently to dimensions where grids become impractical or infeasible. We demonstrate the effectiveness of our approach on several multi-agent collisionavoidance problems in up to 150 dimensions. Furthermore, we empirically observe that the number of parameters in our approach scales linearly with the dimension of the control problem, thereby mitigating the curse of dimensionality.

SympOCnet: Solving optimal control problems with applications to high-dimensional multi-agent path planning problems

“…From the results in Section 5.3, we observe that SympOCnet can handle the path planning problem whose state space has dimension 512, and hence the method can potentially mitigate the CoD. In Section 5.4, we apply SympOCnet to a swarm path planning problem in [74], and demonstrate good performance and efficiency in path planning problems, where the agents move in a three-dimensional space.…”

“…Multiple drones with obstacle avoidance in a three-dimensional space. We consider the three-dimensional swarm path planning example in [74]. To be specific, we consider M = 100 drones with radius 0.18.…”

Deep BSDE-ML Learning and Its Application to Model-Free Optimal Control

“…Recently, deep neural networks have been applied to develop numerical methods that have demonstrated remarkable performance in overcoming the curse of dimensionality and solving high-dimensional HJB equation effectively [2,3,8,10,11,13]. [8] proposes to use traditional numerical methods to evaluate the solution of HJB equation at certain points, and then use these pre-computed data to train a neural network; relying on the generalization of neural networks, the numerical solution on the entire domain is then obtained.…”

“…However, because the nonlinear Feynman-Kac's lemma is involved in the reformulation procedure, Deep BSDE method can only handle some specific PDEs, while DGM is a more general approach. Some other works, e.g., [10] also parameterize the solution of HJB equation with a neural network, but the objective function to be optimized is directly chosen as the cost functional plus some regularization term. And the regularization term is then selected to be the deviation of the trial solution from the PDE and boundary conditions, which is in fact the objective function in DGM.…”

“…[11] solves several HJB equations of a class of nonlinear control systems with Deep BSDE method, but the form of FBSDEs has been modified so that the solution of the forward stochastic differential equation (FSDE) is no longer a trajectory driven by pure noise. To some extent, [10] and [11] can be viewed as the corresponding advanced works of DGM and Deep BSDE in optimal control problems. They no longer consider solving a general PDE, but combine some other properties of HJB eqaution that are inherited from optimal control theory.…”