Learning to Reach Goals via Iterated Supervised Learning

Abstract

Imitation learning algorithms provide a simple and straightforward approach for training control policies via supervised learning. By maximizing the likelihood of good actions provided by an expert demonstrator, supervised imitation learning can produce effective policies without the algorithmic complexities and optimization challenges of reinforcement learning, at the cost of requiring an expert demonstrator to provide the demonstrations. In this paper, we ask: can we take insights from imitation learning to design algorithms that can effectively acquire optimal policies from scratch without any expert demonstrations? The key observation that makes this possible is that, in the multi-task setting, trajectories that are generated by a suboptimal policy can still serve as optimal examples for other tasks. In particular, when tasks correspond to different goals, every trajectory is a successful demonstration for the goal state that it actually reaches. We propose a simple algorithm for learning goal-reaching behaviors without any demonstrations, complicated user-provided reward functions, or complex reinforcement learning methods. Our method simply maximizes the likelihood of actions the agent actually took in its own previous rollouts, conditioned on the goal being the state that it actually reached. Although related variants of this approach have been proposed previously in imitation learning with demonstrations, we show how this approach can effectively learn goal-reaching policies from scratch. We present a theoretical result linking self-supervised imitation learning and reinforcement learning, and empirical results showing that it performs competitively with more complex reinforcement learning methods on a range of challenging goal reaching problems, while yielding advantages in terms of stability and use of offline data.

Decision Transformer: Reinforcement Learning via Sequence Modeling

“…The method proposed by Kumar et al [44] is most similar to our method with K = 1, which we find sequence modeling/long contexts to outperform (see Section 5.3). Ghosh et al [47] extends prior UDRL methods to use state goal conditioning, rather than rewards, and Paster et al [48] further use an LSTM with state goal conditioning for goal-conditoned online RL settings.…”

C-Learning: Horizon-Aware Cumulative Accessibility Estimation

“…For example, Lillicrap et al (2015) extend it to the continuous action space setting, and Fujimoto et al (2018) further stabilize training. These improvements are fully compatible with goal-reaching (Pong et al, 2019;Bharadhwaj et al, 2020a;Ghosh et al, 2019). Andrychowicz et al (2017) proposed Hindsight Experience Replay (HER), which relabels past experience as achieved goals, and allows sample efficient learning from sparse rewards .…”

“…We finish this section by highlighting differences between C-learning and related work. Ghosh et al (2019) proposed GCSL, a method for goal-reaching inspired by supervised learning. In their derivations, they also include a horizon h which their policies can depend on, but they drop this dependence in their experiments as they did not see a practical benefit by including h. We find the opposite for C-learning.…”

“…Multi-goal reinforcement learning tackles the challenging problem of reaching multiple goals, and as a result, is an ideal framework for real-world agents that solve a diverse set of tasks. Despite progress in this field (Kaelbling, 1993;Schaul et al, 2015;Andrychowicz et al, 2017;Ghosh et al, 2019), current algorithms suffer from a set of limitations: an inability to find multiple paths to a goal, high sample complexity, and poor results in complex motion planning tasks. In this paper we propose C-learning, a method which addresses all of these shortcomings.…”

“…The second limitation is sample complexity. Despite significant improvements (Andrychowicz et al, 2017;Ghosh et al, 2019), multi-goal reaching still requires a very large amount of environment interactions for effective learning. We argue that the optimal Q-function must be learned to high accuracy for the agent to achieve reasonable performance, and this leads to sample inefficiency.…”