Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks

Abstract

Offline reinforcement learning leverages previously-collected offline datasets to learn optimal policies with no necessity to access the real environment. Such a paradigm is also desirable for multi-agent reinforcement learning (MARL) tasks, given the increased interactions among agents and with the enviroment. Yet, in MARL, the paradigm of offline pre-training with online fine-tuning has not been studied, nor datasets or benchmarks for offline MARL research are available. In this paper, we facilitate the research by providing large-scale datasets, and use it to examine the usage of the Decision Transformer in the context of MARL. We investigate the generatlisation of MARL offline pre-training in the following three aspects: 1) between single agents and multiple agents, 2) from offline pretraining to the online fine tuning, and 3) to that of multiple downstream tasks with few-shot and zero-shot capabilities. We start by introducing the first offline MARL dataset with diverse quality levels based on the StarCraftII environment, and then propose the novel architecture of multi-agent decision transformer (MADT) for effective offline learning. MADT leverages transformer's modelling ability of sequence modelling and integrates it seamlessly with both offline and online MARL tasks. A crucial benefit of MADT is that it learns generalisable policies that can transfer between different types of agents under different task scenarios. On StarCraft II offline dataset, MADT outperforms the state-of-the-art offline RL baselines. When applied to online tasks, the pre-trained MADT significantly improves sample efficiency, and enjoys strong performance both few-short and zero-shot cases. To our best knowledge, this is the first work that studies and demonstrates the effectiveness of offline pre-trained models in terms of sample efficiency and generalisability enhancements in MARL.

Online Decision Transformer

“…Decision Transformer (DT) (Chen et al, 2021), and the closely related work by Janner et al (2021), provide an alternate perspective by framing offline RL as a sequence modeling problem and solving it via techniques from supervised learning. This provides a simple and scalable framework, including extensions to multi-agent RL (Meng et al, 2021), transfer learning (Boustati et al, 2021), and richer forms of conditioning (Putterman et al; Furuta et al, 2021). We proposed ODT, a simple and robust algorithm for finetuning a pretrained DT in an online setting, thus further expanding its scope to practical scenarios with a mixture of offline and online interaction data.…”

A Survey on Transformers in Reinforcement Learning

“…Step Transformer ConDT [Konan et al, 2022] Offline learned representation conditioning return-dependent transformation SPLT [Villaflor et al, 2022] Offline none min-max search separate models for world and policy ODT [Zheng et al, 2022] Online finetune return-to-go conditioning trajectory-based entropy MADT [Meng et al, 2021] Online finetune (multi-agent) none conditioning separate models for actor and critic Table 1: A summary of Transformers for sequential decision-making.…”

GCS: Graph-based Coordination Strategy for Multi-Agent Reinforcement Learning

“…Deep reinforcement learning has been successfully applied to addressing complex decision problems [13][14][15][16][17]. Due to the widespread existence of multi-agent tasks, MARL has attracted increasing attention, and learning appropriate control policies is important to obtain the maximum cumulative discounted return.…”

“…Let 𝜋 𝑖 represent the fixed policy for agent 𝑖 trained by the graph-based coordinated policy. 𝜑 is the parameter to be solved by the graph generator 𝜌. 𝑔(•) and 𝑑 (•) are the constraint function as shown in Equation (7) and Equation (17). 𝑄 𝜋 𝑖 (•) denotes the state action function.…”