Deep Reactive Policies for Planning in Stochastic Nonlinear Domains

Abstract

Recent advances in applying deep learning to planning have shown that Deep Reactive Policies (DRPs) can be powerful for fast decision-making in complex environments. However, an important limitation of current DRP-based approaches is either the need of optimal planners to be used as ground truth in a supervised learning setting or the sample complexity of high-variance policy gradient estimators, which are particularly troublesome in continuous state-action domains. In order to overcome those limitations, we introduce a framework for training DRPs in continuous stochastic spaces via gradient-based policy search. The general approach is to explicitly encode a parametric policy as a deep neural network, and to formulate the probabilistic planning problem as an optimization task in a stochastic computation graph by exploiting the re-parameterization of the transition probability densities; the optimization is then solved by leveraging gradient descent algorithms that are able to handle non-convex objective functions. We benchmark our approach against stochastic planning domains exhibiting arbitrary differentiable nonlinear transition and cost functions (e.g., Reservoir Control, HVAC and Navigation). Results show that DRPs with more than 125,000 continuous action parameters can be optimized by our approach for problems with 30 state fluents and 30 action fluents on inexpensive hardware under 6 minutes. Also, we observed a speedup of 5 orders of magnitude in the average inference time per decision step of DRPs when compared to other state-of-the-art online gradient-based planners when the same level of solution quality is required.

Planning in stochastic computation graphs: solving stochastic nonlinear problems with backpropagation

“…We focused on continuous stochastic domains with concurrent actions and exogenous events exhibiting nonlinear transition and cost functions. We presented the results published in (Bueno et al, 2019) and showed that training large DRPs with hundred of thousands of continuous action parameters can be carried out within minutes without the need of high-performance hardware. Finally, comparing the DRPs trained by our approach with online state-of-the-art gradient-based planners, we observed a speedup of several orders of magnitude on the time to select actions, which highlights the potential of DRPs for fast decision-making in continuous domains.…”

“…Examples of methods that approximate the original nonlinear planning problem through discretization include Monte-Carlo Tree Search (MCTS) (Chang et al, 2005;Kocsis and Szepesvári, 2006), numeric planning, and Q-learning (Watkins and Dayan, 1992), as well as approaches that resort to first and/or second order approximations such as Symbolic Dynamic Programming (SDP) (Sanner et al, 2011;Vianna et al, 2015;Zamani et al, 2012), MILP-based planning (Say, 2021;Say et al, 2017), and differentiable dynamic programming (DDP) (Jacobson and Mayne, 1970) iLQG (Li and Todorov, 2004) from the optimal control literature. In contrast, methods that are general enough to avoid the need to approximate the original nonlinear problem at the cost of settling for approximate solutions include differentiable planning (i.e., methods based on planning through backpropagation such as TensorPlan (Wu et al, 2017) planner for optimizing plans and Deep Reactive Policies (Bueno et al, 2019) for learning policies parametrized as neural networks in continuous stochastic problems) and model-free policy gradients (Sutton et al, 1999;Williams and Zipser, 1995). Figure 1.2 classifies these approaches according the nature of approximations leveraged in each method.…”

“…In addition, we propose to extend planning through backpropagation to solve offline stochastic planning problems through Deep Reactive Policies trained with policy search (Bueno et al, 2019) over stochastic computation graphs whose stochastic nodes are amenable to reparametrization. Finally, we extend and combine these ideas to implement an online differentiable planner based on information relaxation and anticipatory sampling.…”

“…We conclude the chapter with a number of experiments and a preliminary investigation of the combination of deep reactive policies with online planning that will naturally lead to the developments that shall be presented in the next chapter. We remark that the exposition here is mostly based on our work (Bueno et al, 2019) with some extended discussion and additional experiments.…”

“…In cases where the policy is explicitly represented, the planning task can be typically solved by offline approaches, i.e., an agent uses all the available computational budget to reason and plan before executing any action in the system. Deep Reactive Policies (DRPs) (Bueno et al, 2019) can be learned for problems in which all probability distributions are amenable to be reparametrized thus allowing gradients to flow through the transition model (Chapter 5). We remark that training a DRP via planning through backpropagation is an offline method that requires a priori knowledge of a good policy representation and a fixed planning horizon which are hyperparameters that need to be defined in advance.…”