“…Examples of methods that approximate the original nonlinear planning problem through discretization include Monte-Carlo Tree Search (MCTS) (Chang et al, 2005;Kocsis and Szepesvári, 2006), numeric planning, and Q-learning (Watkins and Dayan, 1992), as well as approaches that resort to first and/or second order approximations such as Symbolic Dynamic Programming (SDP) (Sanner et al, 2011;Vianna et al, 2015;Zamani et al, 2012), MILP-based planning (Say, 2021;Say et al, 2017), and differentiable dynamic programming (DDP) (Jacobson and Mayne, 1970) iLQG (Li and Todorov, 2004) from the optimal control literature. In contrast, methods that are general enough to avoid the need to approximate the original nonlinear problem at the cost of settling for approximate solutions include differentiable planning (i.e., methods based on planning through backpropagation such as TensorPlan (Wu et al, 2017) planner for optimizing plans and Deep Reactive Policies (Bueno et al, 2019) for learning policies parametrized as neural networks in continuous stochastic problems) and model-free policy gradients (Sutton et al, 1999;Williams and Zipser, 1995). Figure 1.2 classifies these approaches according the nature of approximations leveraged in each method.…”