Offline Decentralized Multi-Agent Reinforcement Learning

Jiang, Jiechuan; Lu, Zongqing

doi:10.48550/arxiv.2108.01832

Cited by 9 publications

(16 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In many practical scenarios, we only have access to the offline data or it is too expensive to frequently change the policy [Zhang et al, 2021a]. While there are plenty of empirical works on offline MARL [Pan et al, 2021, Jiang andLu, 2021], the theoretical understanding about offline MARL is still very limited. In this work, we take an initial step towards understanding when offline MARL is provably solvable.…”

Section: Introductionmentioning

confidence: 99%

When is Offline Two-Player Zero-Sum Markov Game Solvable?

Cui¹,

Du²

2022

Preprint

View full text Add to dashboard Cite

We study what dataset assumption permits solving offline two-player zero-sum Markov game. In stark contrast to the offline single-agent Markov decision process, we show that the single strategy concentration assumption is insufficient for learning the Nash equilibrium (NE) strategy in offline two-player zero-sum Markov games. On the other hand, we propose a new assumption named unilateral concentration and design a pessimism-type algorithm that is provably efficient under this assumption. In addition, we show that the unilateral concentration assumption is necessary for learning an NE strategy. Furthermore, our algorithm can achieve minimax sample complexity without any modification for two widely studied settings: dataset with uniform concentration assumption and turn-based Markov game. Our work serves as an important initial step towards understanding offline multi-agent reinforcement learning. * While we assume deterministic rewards for simplicity, our results can be straightforwardly generalized to unknown stochastic rewards, as the major difficulty is in learning the transitions rather than learning the rewards.† Stochastic initial state is equivalent to an MDP with deterministic initial state by creating a dummy initial state which transit to the next state following that initial state distribution.

show abstract

Section: Introductionmentioning

confidence: 99%

When is Offline Two-Player Zero-Sum Markov Game Solvable?

Cui¹,

Du²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Substantial works presented in offline RL aim at resolving the distribution shift between the static offline datasets and the online environment interactions Fujimoto et al, 2019;Kumar et al, 2019). Particularly, ; Jiang & Lu (2021) constrain off-policy algorithms in offline MARL. Related to our work for the improvement of sample efficiency, Nair et al (2020) derives the KKT condition of the online objective generating a advantage weight to avoid OOD problem.…”

Section: Related Workmentioning

confidence: 99%

Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks

Meng¹,

Wen²,

Yang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Offline reinforcement learning leverages previously-collected offline datasets to learn optimal policies with no necessity to access the real environment. Such a paradigm is also desirable for multi-agent reinforcement learning (MARL) tasks, given the increased interactions among agents and with the enviroment. Yet, in MARL, the paradigm of offline pre-training with online fine-tuning has not been studied, nor datasets or benchmarks for offline MARL research are available. In this paper, we facilitate the research by providing large-scale datasets, and use it to examine the usage of the Decision Transformer in the context of MARL. We investigate the generatlisation of MARL offline pre-training in the following three aspects: 1) between single agents and multiple agents, 2) from offline pretraining to the online fine tuning, and 3) to that of multiple downstream tasks with few-shot and zero-shot capabilities. We start by introducing the first offline MARL dataset with diverse quality levels based on the StarCraftII environment, and then propose the novel architecture of multi-agent decision transformer (MADT) for effective offline learning. MADT leverages transformer's modelling ability of sequence modelling and integrates it seamlessly with both offline and online MARL tasks. A crucial benefit of MADT is that it learns generalisable policies that can transfer between different types of agents under different task scenarios. On StarCraft II offline dataset, MADT outperforms the state-of-the-art offline RL baselines. When applied to online tasks, the pre-trained MADT significantly improves sample efficiency, and enjoys strong performance both few-short and zero-shot cases. To our best knowledge, this is the first work that studies and demonstrates the effectiveness of offline pre-trained models in terms of sample efficiency and generalisability enhancements in MARL.

show abstract

“…In addition, it presents a critical challenge in the decentralized setting when the datasets for each agent only consist of its own action instead of the joint action [19]. Jiang and Lu [19] address the challenges based on the behavior regularization BCQ [13] algorithm while Yang et al [57] propose to estimate the target value based on the next action from the dataset. As a result, both methods largely depend on the quality of the dataset.…”

Section: Multi-agent Reinforcement Learningmentioning

confidence: 99%

“…However, many practical scenarios involve multiple agents, e.g., multi-robot control [4], autonomous driving [41,45]. Therefore, offline multi-agent reinforcement learning (MARL) [19,57] is crucial for solving real-world tasks. Observing recent success of Independent PPO [8] and Multi-Agent PPO [58], both of which are based on the PPO [49] algorithm, we find that online RL algorithms can be transferred to multi-agent scenarios through either decentralized training or a centralized value function without bells and whistles.…”

Section: Introductionmentioning

confidence: 99%

Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification

Pan¹,

Huang²,

Ma³

et al. 2021

Preprint

View full text Add to dashboard Cite

The idea of conservatism has led to significant progress in offline reinforcement learning (RL) where an agent learns from pre-collected datasets. However, it is still an open question to resolve offline RL in the more practical multi-agent setting as many real-world scenarios involve interaction among multiple agents. Given the recent success of transferring online RL algorithms to the multi-agent setting, one may expect that offline RL algorithms will also transfer to multi-agent settings directly. Surprisingly, when conservatism-based algorithms are applied to the multi-agent setting, the performance degrades significantly with an increasing number of agents. Towards mitigating the degradation, we identify that a key issue that the landscape of the value function can be non-concave and policy gradient improvements are prone to local optima. Multiple agents exacerbate the problem since the suboptimal policy by any agent could lead to uncoordinated global failure. Following this intuition, we propose a simple yet effective method, Offline Multi-Agent RL with Actor Rectification (OMAR), to tackle this critical challenge via an effective combination of first-order policy gradient and zeroth-order optimization methods for the actor to better optimize the conservative value function. Despite the simplicity, OMAR significantly outperforms strong baselines with state-of-the-art performance in multi-agent continuous control benchmarks.

show abstract

Offline Decentralized Multi-Agent Reinforcement Learning

Cited by 9 publications

References 9 publications

When is Offline Two-Player Zero-Sum Markov Game Solvable?

When is Offline Two-Player Zero-Sum Markov Game Solvable?

Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks

Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification

Contact Info

Product

Resources

About