When Can We Learn General-Sum Markov Games with a Large Number of Players Sample-Efficiently?

Song, Ziang; Song, Mei; Bai, Yu

doi:10.48550/arxiv.2110.04184

Cited by 18 publications

(32 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…V-learning-initially coupled with the FTRL algorithm as adversarial bandit subroutine-is firstly proposed in the conference version of this paper [3], for finding Nash equilibria in the two-player zero-sum setting. During the preparation of this draft, we note two very recent independent works [47,32], whose results partially overlap with the results of this paper in the multiplayer general-sum setting. In particular, Mao and Bas ¸ar [32] use V-learning with stablized online mirror descent as adversarial bandit subroutine, and learn ǫ-CCE in O(H 6 SA/ǫ 2 ) episodes, where A = max j∈[m] A j .…”

Section: Related Worksupporting

confidence: 66%

See 1 more Smart Citation

V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RL

Jin¹,

Liu²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

A major challenge of multiagent reinforcement learning (MARL) is the curse of multiagents, where the size of the joint action space scales exponentially with the number of agents. This remains to be a bottleneck for designing efficient MARL algorithms even in a basic scenario with finitely many states and actions. This paper resolves this challenge for the model of episodic Markov games. We design a new class of fully decentralized algorithms-V-learning, which provably learns Nash equilibria (in the two-player zero-sum setting), correlated equilibria and coarse correlated equilibria (in the multiplayer general-sum setting) in a number of samples that only scales with max i∈[m] A i , where A i is the number of actions for the i th player. This is in sharp contrast to the size of the joint action space which is m i=1 A i . V-learning (in its basic form) is a new class of single-agent RL algorithms that convert any adversarial bandit algorithm with suitable regret guarantees into a RL algorithm. Similar to the classical Q-learning algorithm, it performs incremental updates to the value functions. Different from Q-learning, it only maintains the estimates of V-values instead of Q-values. This key difference allows V-learning to achieve the claimed guarantees in the MARL setting by simply letting all agents run V-learning independently.

show abstract

Section: Related Worksupporting

confidence: 66%

“…This is one H factor larger than what is required in Theorem 6 of this paper. Song et al [47] considers similar V-learning style algorithms for learning both ǫ-CCE and ǫ-CE. For the latter objective, they require O(H 6 SA 2 /ǫ 2 ) episodes which is again one H factor larger than what is required in Theorem 7 of this paper.…”

Section: Related Workmentioning

confidence: 99%

V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RL

Jin¹,

Liu²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Zero-sum Markov game has been widely studied since the seminal work [Shapley, 1953]. When the transition kernel is unknown, different sampling oracles are utilized to acquire samples, including online sampling , Xie et al, 2020a, Liu et al, 2021, Jin et al, 2021a, Song et al, 2021, generative model sampling [Sidford et al, 2020, Cui and Yang, 2020, Zhang et al, 2020, Jia et al, 2019. For offline sampling oracle, Zhang et al [2021b] provides finite sample bound for a decentralized algorithm with network communication under uniform concentration assumption and Abe and Kaneko [2020] considers offline policy evaluation, again under the uniform concentration assumption.…”

Section: Related Workmentioning

confidence: 99%

When is Offline Two-Player Zero-Sum Markov Game Solvable?

Cui¹,

Du²

2022

Preprint

View full text Add to dashboard Cite

We study what dataset assumption permits solving offline two-player zero-sum Markov game. In stark contrast to the offline single-agent Markov decision process, we show that the single strategy concentration assumption is insufficient for learning the Nash equilibrium (NE) strategy in offline two-player zero-sum Markov games. On the other hand, we propose a new assumption named unilateral concentration and design a pessimism-type algorithm that is provably efficient under this assumption. In addition, we show that the unilateral concentration assumption is necessary for learning an NE strategy. Furthermore, our algorithm can achieve minimax sample complexity without any modification for two widely studied settings: dataset with uniform concentration assumption and turn-based Markov game. Our work serves as an important initial step towards understanding offline multi-agent reinforcement learning. * While we assume deterministic rewards for simplicity, our results can be straightforwardly generalized to unknown stochastic rewards, as the major difficulty is in learning the transitions rather than learning the rewards.† Stochastic initial state is equivalent to an MDP with deterministic initial state by creating a dummy initial state which transit to the next state following that initial state distribution.

show abstract

“…However, the results for equilibrium computation in stochastic games are often much weaker than those for single-agent MDPs, mainly because of the nonstationary environment that is induced by players' decisions [14]. In general, there are strong lower bounds for computing stationary NE in stochastic games, which grows exponentially in terms of the number of players [20]. Therefore, prior work has largely focused on the special case of two-player zero-sum stochastic games [14], [21]- [23].…”

mentioning

confidence: 99%

Learning Stationary Nash Equilibrium Policies in $n$-Player Stochastic Games with Independent Chains

Etesami¹

2022

Preprint

View full text Add to dashboard Cite

We consider a subclass of n-player stochastic games, in which players have their own internal state/action spaces while they are coupled through their payoff functions. It is assumed that players' internal chains are driven by independent transition probabilities. Moreover, players can only receive realizations of their payoffs but not the actual functions, nor can they observe each others' states/actions. Under some assumptions on the structure of the payoff functions, we develop efficient learning algorithms based on Dual Averaging and Dual Mirror Descent, which provably converge almost surely or in expectation to the set of ǫ-Nash equilibrium policies. In particular, we derive upper bounds on the number of iterates that scale polynomially in terms of the game parameters to achieve an ǫ-Nash equilibrium policy. Besides Markov potential games and linear-quadratic stochastic games, this work provides another interesting subclass of n-player stochastic games that provably admit polynomial-time learning algorithm for finding their ǫ-Nash equilibrium policies. Index TermsStochastic games; stationary Nash equilibrium, dual averaging, dual mirror descent, learning in games. I. INTRODUCTIONSince the early work on the existence of a mixed-strategy Nash equilibrium in static noncooperative games [1], and its extension on the existence of stationary Nash equilibrium policies in dynamic stochastic games [2], substantial research has been done to develop scalable algorithms for computing Nash equilibrium (NE) points in static and dynamic environments. NE provides a stable solution concept for strategic multiagent decision-making systems, which is a desirable property in many applications such as socioeconomic systems [3], network security [4], routing and scheduling [5], among many others [6], [7].Unfortunately, computing NE is generally PPAD-hard [8], and it is unlikely to admit a polynomial-time algorithm. Thus, to overcome this fundamental barrier, two main approaches have been adapted in the past literature: i) searching for relaxed notions of stable solutions such as correlated equilibrium [9], which includes the set of NE, and ii) searching for NE points in special structured games such as potential games [10], or concave games [11]. Thanks to recent advances in the field of learning theory, it is known that some tailored algorithms for finding relaxed notions of equilibrium in case (i) can also be used to compute NE points of structured games in case (ii). For instance, it is known that the so-called no-regret algorithms always converge to the set of coarse correlated equilibria [7], and they can also be used to compute NE in the class of socially concave games [12]. However, such results are mainly developed for static games, in which players repeatedly play the same game and gradually learn the underlying stationary environment. Unfortunately, extensions of such results to dynamic stochastic games [2], [13], in which the state of the game evolves as a result of players' past decisions and the realizations of a stoc...

show abstract

When Can We Learn General-Sum Markov Games with a Large Number of Players Sample-Efficiently?

Cited by 18 publications

References 42 publications

V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RL

V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RL

When is Offline Two-Player Zero-Sum Markov Game Solvable?

Learning Stationary Nash Equilibrium Policies in $n$-Player Stochastic Games with Independent Chains

Contact Info

Product

Resources

About