Online Decision Transformer

Zheng, Qinqing; Zhang, Amy; Grover, Aditya

doi:10.48550/arxiv.2202.05607

Cited by 12 publications

(18 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The most closely related architectures to that of Gato are Decision Transformers (Chen et al, 2021b;Furuta et al, 2021;Reid et al, 2022;Zheng et al, 2022) and Trajectory Transformer (Janner et al, 2021), which showed the usefulness of highly generic LM-like architectures for a variety of control problems. Gato also uses an LM-like architecture for control, but with design differences chosen to support multi-modality, multi-embodiment, large scale and general purpose deployment.…”

Section: Related Workmentioning

confidence: 99%

“…This is also true of state-of-the-art RL methods applied to board games (Schrittwieser et al, 2020). Moreover, this choice has been adopted by off-line RL benchmarks (Fu et al, 2020; and recent works on large sequence neural networks for control, including decision transformers (Chen et al, 2021b;Reid et al, 2022;Zheng et al, 2022) and the Trajectory Transformer of Janner et al (2021). In contrast, in this work we learn a single network with the same weights across a diverse set of tasks.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

A Generalist Agent

Reed¹,

Żołna²,

Parisotto³

et al. 2022

Preprint

View full text Add to dashboard Cite

Inspired by progress in large-scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens. In this report we describe the model and the data, and document the current capabilities of Gato.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

A Generalist Agent

Reed¹,

Żołna²,

Parisotto³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Furthermore, we provide comparisons against existing behavioral cloning, online and offline RL methods, and contrastive representations [77,54]. Other works that also consider LLM-like sequence modeling for a variety of single control tasks include [63,81,34,22,55].…”

Section: Related Workmentioning

confidence: 99%

Multi-Game Decision Transformers

Lee¹,

Nachum²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

A longstanding goal of the field of AI is a strategy for compiling diverse experience into a highly capable, generalist agent. In the subfields of vision and language, this was largely achieved by scaling up transformer-based models and training them on large, diverse datasets. Motivated by this progress, we investigate whether the same strategy can be used to produce generalist reinforcement learning agents. Specifically, we show that a single transformer-based model -with a single set of weights -trained purely offline can play a suite of up to 46 Atari games simultaneously at close-to-human performance. When trained and evaluated appropriately, we find that the same trends observed in language and vision hold, including scaling of performance with model size and rapid adaptation to new games via fine-tuning. We compare several approaches in this multi-game setting, such as online and offline RL methods and behavioral cloning, and find that our Multi-Game Decision Transformer models offer the best scalability and performance. We release the pre-trained models and code to encourage further research in this direction. 1

show abstract

“…The chain of thought imitation learning problem we formulate is domain agnostic and applicable to many sequential decision making task traditionally solved by imitation learning in a Markovian setting such as robot locomotion, navigation, manipulation, and strategy games. Unlike language-based tasks, problems in decision making have only recently started being explored by language models [68,69,70,71,72,73], as the Markovian nature of these problems brings the value of sequence modeling into question. Our work contributes to bridging the gap between learning memoryless policies in Markovian environments and the intuition that large sequence models should help in reasoning-based decision making.…”

Section: Related Workmentioning

confidence: 99%

Chain of Thought Imitation with Procedure Cloning

Yang¹,

Schuurmans²,

Abbeel³

et al. 2022

Preprint

View full text Add to dashboard Cite

Imitation learning aims to extract high-performance policies from logged demonstrations of expert behavior. It is common to frame imitation learning as a supervised learning problem in which one fits a function approximator to the input-output mapping exhibited by the logged demonstrations (input observations to output actions). While the framing of imitation learning as a supervised input-output learning problem allows for applicability in a wide variety of settings, it is also an overly simplistic view of the problem in situations where the expert demonstrations provide much richer insight into expert behavior. For example, applications such as path navigation, robot manipulation, and strategy games acquire expert demonstrations via planning, search, or some other multi-step algorithm, revealing not just the output action to be imitated but also the procedure for how to determine this action. While these intermediate computations may use tools not available to the agent during inference (e.g., environment simulators), they are nevertheless informative as a way to explain an expert's mapping of state to actions. To properly leverage expert procedure information without relying on the privileged tools the expert may have used to perform the procedure, we propose procedure cloning, which applies supervised sequence prediction to imitate the series of expert computations. This way, procedure cloning learns not only what to do (i.e., the output action), but how and why to do it (i.e., the procedure). Through empirical analysis on navigation, simulated robotic manipulation, and game-playing environments, we show that imitating the intermediate computations of an expert's behavior enables procedure cloning to learn policies exhibiting significant generalization to unseen environment configurations, including those configurations for which running the expert's procedure directly is infeasible. 1

show abstract

Online Decision Transformer

Cited by 12 publications

References 20 publications

A Generalist Agent

A Generalist Agent

Multi-Game Decision Transformers

Chain of Thought Imitation with Procedure Cloning

Contact Info

Product

Resources

About