Hierarchical Neural Dynamic Policies

Abstract

We tackle the problem of generalization to unseen configurations for dynamic tasks in the real world while learning from high-dimensional image input. The family of nonlinear dynamical system-based methods have successfully demonstrated dynamic robot behaviors but have difficulty in generalizing to unseen configurations as well as learning from image inputs. Recent works approach this issue by using deep network policies and reparameterize actions to embed the structure of dynamical systems but still struggle in domains with diverse configurations of image goals, and hence, find it difficult to generalize. In this paper, we address this dichotomy by leveraging embedding the structure of dynamical systems in a hierarchical deep policy learning framework, called Hierarchical Neural Dynamical Policies (H-NDPs). Instead of fitting deep dynamical systems to diverse data directly, H-NDPs form a curriculum by learning local dynamical system-based policies on small regions in state-space and then distill them into a global dynamical system-based policy that operates only from high-dimensional images. H-NDPs additionally provide smooth trajectories, a strong safety benefit in the real world. We perform extensive experiments on dynamic tasks both in the real world (digit writing, scooping, and pouring) and simulation (catching, throwing, picking). We show that H-NDPs are easily integrated with both imitation as well as reinforcement learning setups and achieve state-of-the-art results. Video results are at https://shikharbahl.github.io/hierarchical-ndps/.

Guided Reinforcement Learning: A Review and Evaluation for Efficient and Effective Real-World Robotics [Survey]

“…State representation [24], [53], [86], [87], [97] [22], [55], [138] Reward design [38], [119] [18] [93] [135] [23], [31], [33], [54] Abstract learning [27] [106], [107] [3], [16], [82], [134], [136] Offline RL [26] [1], [20], [39], [63], [116], [133], [140] Parallel learning [48], [114] [11], [32], [44], [58], [79], [80], [88], [113] Learning from demonstration [7], [35] [19]…”

“…Other approaches first train a teacher policy on unchanging task setups via, e.g., RL and distill a policy capable of interpolating among different task setups, such as [7], which chooses neural dynamical policies [8] to present the teacher and students. Others learn teacher policies on true state information to then derive a student policy conditioned on a reduced or substituted input space, e.g., Chen et al [21], whose final vision-based policies are able to reorient objects in the shadow hand domain, and Lee et al [68], who also distill a vision-based policy and test it in their red-green-blue-stacking benchmark.…”

“…Second, simplifying the learning task by means of task-specific action spaces and hybrid modelbased and model-free approaches can improve the overall efficiency. Finally, training based on expert demonstrations tends to be rich in information and hence can accelerate policy training [7], [124], [131]. Furthermore, the efficiency can likely be improved by employing more instructive state representations, applying a curriculum to gradually tackle difficult learning tasks, and utilizing accurate simulation environments.…”

“…Using Multiple Guided RL Methods First, we find that the guided RL-compliant papers tend to use a variety of guided RL methods. For instance, [7], [27], and [51] utilize at least three guided RL approaches, while [43], [70], and [82] deploy five or more approaches to obtain improvements in all three guided RL dimensions.…”

Learning Pneumatic Non-Prehensile Manipulation with a Mobile Blower

“…Within hierarchical RL, there also exists a class of compositional methods in which a high-level policy issues commands to be executed by a low-level policy. The commands can be specified by learned latent representations [32], [33], [34]), low-level control parameters [35], [36], [37], [38], or subgoals, which can be hand-designed [39], task-specific locations [40], [41], [42], [43], [44], [45], [46], target object states [47], [48], [49]), or target states in a learned latent space [50], [51], [52]. In contrast to these works, our multi-frequency system uses no explicit skills, commands, or subgoals.…”