The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces

Jin, Chi; Liu, Qinghua; Yu, Tiancheng

doi:10.48550/arxiv.2106.03352

Cited by 11 publications

(26 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, we focus our attention on theoretical results for the tabular setting, where the numbers of states and actions are finite. We acknowledge that there has been much recent work in RL for continuous state spaces [see, e.g., 21,23,56,24,55,25], but this setting is beyond our scope.…”

Section: Related Workmentioning

confidence: 99%

V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RL

Jin¹,

Liu²,

Wang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

A major challenge of multiagent reinforcement learning (MARL) is the curse of multiagents, where the size of the joint action space scales exponentially with the number of agents. This remains to be a bottleneck for designing efficient MARL algorithms even in a basic scenario with finitely many states and actions. This paper resolves this challenge for the model of episodic Markov games. We design a new class of fully decentralized algorithms-V-learning, which provably learns Nash equilibria (in the two-player zero-sum setting), correlated equilibria and coarse correlated equilibria (in the multiplayer general-sum setting) in a number of samples that only scales with max i∈[m] A i , where A i is the number of actions for the i th player. This is in sharp contrast to the size of the joint action space which is m i=1 A i . V-learning (in its basic form) is a new class of single-agent RL algorithms that convert any adversarial bandit algorithm with suitable regret guarantees into a RL algorithm. Similar to the classical Q-learning algorithm, it performs incremental updates to the value functions. Different from Q-learning, it only maintains the estimates of V-values instead of Q-values. This key difference allows V-learning to achieve the claimed guarantees in the MARL setting by simply letting all agents run V-learning independently.

show abstract

Section: Related Workmentioning

confidence: 99%

V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RL

Jin¹,

Liu²,

Wang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Most of these work focus on tabular setting [WHL17, PSPP17, BJ20, BJWX21, BJY20, ZKBY20, ZTLD21] or linear function approximation settings [XCWY20]. Moreover, the concurrent work of [JLY21] studies competitive RL with general function approximation. In particular, under the decoupled setting where the learner only controls P1, [JLY21] proposes a provably sample efficient and model-free algorithm based on successive hypothesis elimination and the principle of optimism in the face of uncertainty [LS18].…”

Section: Related Workmentioning

confidence: 99%

“…Moreover, the concurrent work of [JLY21] studies competitive RL with general function approximation. In particular, under the decoupled setting where the learner only controls P1, [JLY21] proposes a provably sample efficient and model-free algorithm based on successive hypothesis elimination and the principle of optimism in the face of uncertainty [LS18]. The proposed algorithm achieves a sublinear O( √ K) regret for MDPs with low minimax Eluder dimensions.…”

Section: Related Workmentioning

confidence: 99%

Towards General Function Approximation in Zero-Sum Markov Games

Huang

Lee

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

This paper considers two-player zero-sum finite-horizon Markov games with simultaneous moves. The study focuses on the challenging settings where the value function or the model is parameterized by general function classes. Provably efficient algorithms for both decoupled and coordinated settings are developed. In the decoupled setting where the agent controls a single player and plays against an arbitrary opponent, we propose a new model-free algorithm. The sample complexity is governed by the Minimax Eluder dimension-a new dimension of the function class in Markov games. As a special case, this method improves the state-of-the-art algorithm by a √ d factor in the regret when the reward function and transition kernel are parameterized with d-dimensional linear features. In the coordinated setting where both players are controlled by the agent, we propose a model-based algorithm and a modelfree algorithm. In the model-based algorithm, we prove that sample complexity can be bounded by a generalization of Witness rank to Markov games. The model-free algorithm enjoys a √ K-regret upper bound where K is the number of episodes. Our algorithms are based on new techniques of alternate optimism.

show abstract

“…In the online setting where the opponent is arbitrary, (Xie et al 2020;Jin, Liu, and Yu 2021) achieve a regret bound of O( √ T ) in the finite-horizon SGs with linear and general function approximation, respectively. However, in the applications where the interaction between the players and the environment is non-stopping (e.g., stock trading), the infinitehorizon SG is more suitable.…”

Section: Related Literaturementioning

confidence: 99%

Learning Zero-sum Stochastic Games with Posterior Sampling

Jafarnia-Jahromi,

Jain,

Nayyar

2021

Preprint

View full text Add to dashboard Cite

In this paper, we propose Posterior Sampling Reinforcement Learning for Zero-sum Stochastic Games (PSRL-ZSG), the first online learning algorithm that achieves Bayesian regret bound of O(HS √ AT ) in the infinite-horizon zero-sum stochastic games with average-reward criterion. Here H is an upper bound on the span of the bias function, S is the number of states, A is the number of joint actions and T is the horizon. We consider the online setting where the opponent can not be controlled and can take any arbitrary timeadaptive history-dependent strategy. This improves the best existing regret bound of O( 3 √ DS 2 AT 2 ) by (Wei, Hong, and Lu 2017) under the same assumption and matches the theoretical lower bound in A and T .

show abstract

The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces

Cited by 11 publications

References 35 publications

V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RL

V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RL

Towards General Function Approximation in Zero-Sum Markov Games

Learning Zero-sum Stochastic Games with Posterior Sampling

Contact Info

Product

Resources

About