Compositional Transfer in Hierarchical Reinforcement Learning

Abstract

The successful application of general reinforcement learning algorithms to real-world robotics applications is often limited by their high data requirements. We introduce Regu larized Hierarchical Policy Optimization (RHPO) to improve data-efliciency for domains with multiple dominant tasks and ultimately reduce required platform time. To this end, we employ compositional inductive biases on multiple levels and corresponding mechanisms for sharing off-policy transition data across low-level controllers and tasks as well as scheduling of tasks. The presented algorithm enables stable and fast learning for complex, real-world domains in the parallel multitask and sequential transfer case. We show that the investigated types of hierarchy enable positive transfer while partially mitigating negative interference and evaluate the benefits of additional incentives for efficient, compositional task solutions in single task domains. Finally, we demonstrate substantial data-efficiency and final performance gains over competitive baselines in a week-long, physical robot stacking experiment.

Priors, Hierarchy, and Information Asymmetry for Skill Transfer in Reinforcement Learning

“…This promotes reduced information conditioning for π L than π 0 . Thus, in-line with previous works [14,9], we share the low-level policy π L between policy and prior, and condition only on state s t , ensuring minimal covariate shift and the discovery of instantaneous behaviours that generalise favourably. Unlike prior works, we consider conditioning the high-level prior, π H 0 , on additional information enabling richer skill transfer.…”

“…By presenting multiple priors, we enable a comparison with existing literature [14,19,20,21]. With the right masking, one can recover previously investigated asymmetries [14,19], explore additional ones, and also express purely hierarchical [9] and KL-regularized equivalents [8].…”

“…RecSAC represents a historydependent SAC [27], trained directly on the transfer task and without prior experience. Hier-RecSAC, represents a hierarchical-equivalent of RecSAC using same hierarchical decomposition as APES, and is also trained directly on the task at hand, and is akin to the method proposed in Wulfmeier et al [9]. Table 3a compares performance of all approaches on the transfer tasks.…”

“…By regularizing policy behaviour against a learnt task-agnostic prior, common behaviours across tasks are distilled into the prior, which encourages their reuse. Concurrently, hierarchical RL also enables skill discovery [9,10,11,12,13] by considering a two-level hierarchy in which the high-level policy is task-conditioned, whilst the low-level remains task-agnostic. The lower level of the hierarchy therefore also discovers skills that are transferable across tasks.…”

“…Therefore, IA crucially biases learnt behaviour and how it transfers across environments. Previous works have motivated their chosen IAs primarily on intuition [14,8,9], which, if sub-optimal, limits transfer benefits. We demonstrate that this indeed is the case for the hierarchical KL-regularized method in [14].…”