Generalized Decision Transformer for Offline Hindsight Information Matching

Abstract

How to extract as much learning signal from each trajectory data has been a key problem in reinforcement learning (RL), where sample inefficiency has posed serious challenges for practical applications. Recent works have shown that using expressive policy function approximators and conditioning on future trajectory information -such as future states in hindsight experience replay (HER) or returnsto-go in Decision Transformer (DT) -enables efficient learning of multi-task policies, where at times online RL is fully replaced by offline behavioral cloning (BC), e.g. sequence modeling. We demonstrate that all these approaches are doing hindsight information matching (HIM) -training policies that can output the rest of trajectory that matches some statistics of future state information. We present Generalized Decision Transformer (GDT) for solving any HIM problem, and show how different choices for the feature function and the anti-causal aggregator not only recover DT as a special case, but also lead to novel Categorical DT (CDT) and Bi-directional DT (BDT) for matching different statistics of the future. For evaluating CDT and BDT, we define offline multi-task state-marginal matching (SMM) and imitation learning (IL) as two generic HIM problems, propose a Wasserstein distance loss as a metric for both, and empirically study them on MuJoCo continuous control benchmarks. Categorical DT, which simply replaces anti-causal summation with anti-causal binning in DT, enables arguably the first effective offline multi-task SMM algorithm that generalizes well to unseen (and even synthetic) multi-modal reward or state-feature distributions. Bi-directional DT, which uses an anti-causal second transformer as the aggregator, can learn to model any statistics of the future and outperforms DT variants in offline multi-task IL, i.e. one-shot IL. Our generalized formulations from HIM and GDT greatly expand the role of powerful sequence modeling architectures in modern RL.

Can Wikipedia Help Offline Reinforcement Learning?

“…Numerous algorithmic work ensued (Wu et al, 2019;Jaques et al, 2020;Ghasemipour et al, 2021;Kumar et al, 2020;Fujimoto & Gu, 2021) with various applications (Jaques et al, 2020;Chebotar et al, 2021). Building on reward-conditioned imitation learning (Srivastava et al, 2019;Kumar et al, 2019), Transformer architecture has been recently adopted for replacing offline RL with sequence modeling (Chen et al, 2021;Janner et al, 2021;Furuta et al, 2021). Despite initial successes, many techniques popular in language modeling have yet to be experimented in these offline RL benchmarks, and our work constitutes an initial step toward bridging the two communities.…”

“…Concurrently, offline reinforcement learning (RL) has been seen as analogous to sequence modeling (Chen et al, 2021;Janner et al, 2021;Furuta et al, 2021), framed as simply supervised learning to fit return-augmented trajectories in an offline dataset. This relaxation, doing away with many of the complexities commonly associated with reinforcement learning (Watkins & Dayan, 1992;Kakade, 2001), allows us to take advantage of techniques popularized in sequence modeling tasks for RL.…”

“…However, such concept is still relatively fresh in RL (Singh et al, 2020;Tirumala et al, 2020), due to the difficulty in parameterizing different scenes and tasks through a single network (Wang et al, 2018b;Jiang et al, 2019;Zeng et al, 2020) as well as the lack of large off-the-shelf datasets for pre-training (Cobbe et al, 2020;Zhu et al, 2020;Yu et al, 2020). Adopting pre-training as a default option for recent Transformer-based methods (Chen et al, 2021;Janner et al, 2021;Furuta et al, 2021) appears far away -if we only look within RL.…”

“…Common approaches include value-based or model-based objectives with regularization (Fujimoto et al, 2019;Levine et al, 2020), and more recently, direct generative modeling of these trajectories conditioned on hindsight returns (Chen et al, 2021;Janner et al, 2021;Furuta et al, 2021).…”

Online Decision Transformer

“…Decision Transformer (DT) (Chen et al, 2021), and the closely related work by Janner et al (2021), provide an alternate perspective by framing offline RL as a sequence modeling problem and solving it via techniques from supervised learning. This provides a simple and scalable framework, including extensions to multi-agent RL (Meng et al, 2021), transfer learning (Boustati et al, 2021), and richer forms of conditioning (Putterman et al; Furuta et al, 2021). We proposed ODT, a simple and robust algorithm for finetuning a pretrained DT in an online setting, thus further expanding its scope to practical scenarios with a mixture of offline and online interaction data.…”