Learning to Plan from Raw Data in Grid-based Games

Abstract

An agent that autonomously learns to act in its environment must acquire a model of the domain dynamics. This can be a challenging task, especially in real-world domains, where observations are high-dimensional and noisy. Although in automated planning the dynamics are typically given, there are action schema learning approaches that learn sym- bolic rules (e.g. STRIPS or PDDL) to be used by traditional planners. However, these algorithms rely on logical descriptions of environment observations. In contrast, recent methods in deep reinforcement learning for games learn from pixel observations. However, they typically do not acquire an environment model, but a policy for one-step action selec- tion. Even when a model is learned, it cannot generalize to unseen instances of the training domain. Here we propose a neural network-based method that learns from visual obser- vations an approximate, compact, implicit representation of the domain dynamics, which can be used for planning with standard search algorithms, and generalizes to novel domain instances. The learned model is composed of submodules, each implicitly representing an action schema in the traditional sense. We evaluate our approach on visual versions of the standard domain Sokoban, and show that, by training on one single instance, it learns a transition model that can be successfully used to solve new levels of the game.

SPOTTER: Extending Symbolic Planning Operators through Targeted Reinforcement Learning

“…The experimental results show that SPOTTER achieved higher overall rewards than the baselines in the given time frame and did so more quickly. 7 Crucially, the agent learned the missing operator for moving the blue ball out of the way in Level 2, and was immediately able to use this operator in Level 3. This is demonstrated both by the fact that the agent did not experience any drop in performance when transitioning to Level 3 and also we know from running the experiment that the agent did not enter learn or gen-precon in Level 3.…”

“…Accordingly, we did not compare against any deep RL baselines. We also did not compare transfer learning and curriculum learning approaches as these approaches 7 Code implementing SPOTTER and the baselines along with experiments will be made available post-review. 8 In the supplementary material, we provide the learned operator described in PDDL, learning curves for the baselines over 2,000,000 episodes, and videos showing SPOT-TER's integrated planning and learning.…”

“…The examples consist of successful fluent state and operator pairs corresponding to a sequence of transitions in the symbolic domain [35]. Subsequent work has explored how domains can be learned with partial knowledge of successful traces [2,6], and with neural networks capable of approximating from partial traces [33] and learning models from pixels [4,7].…”

Learning to select goals in Automated Planning with Deep-Q Learning

“…Neural networks have also been applied to other aspects of planning. For instance, (Dittadi et al, 2018) trains a NN that learns a planning domain just from visual observations, assuming that actions have local preconditions and effects. The learned domain is generalizable across different problems of the same domain and, thus, can be used by a planner to solve these problems.…”

Goal Reasoning by Selecting Subgoals with Deep Q-Learning

“…Neural networks have also been applied to other aspects of planning. For instance, (Dittadi, Bolander, and Winther 2018) trains a NN that learns a planning domain just from visual observations, assuming that actions have local preconditions and effects. The learnt domain is generalizable across different problems of the same domain and, thus, can be used by a planner to solve these problems.…”