Online Reinforcement Learning in Stochastic Games

Wei, Chen-Yu; Hong, Yi-Te; Lu, Chi-Jen

doi:10.48550/arxiv.1712.00579

Cited by 17 publications

(7 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Markov Game (MG), also known as stochastic game [42], is a popular model in multi-agent RL [32]. Early works have mainly focused on finding Nash equilibria of MGs with known transition and reward [33,21,20,53], or under strong reachability conditions such as simulators [52,23,43,63,54] where exploration is not needed. A recent line of works provide non-asymptotic guarantees for learning two-player zero-sum tabular MGs, in the setting that requires strategic exploration.…”

Section: Introductionmentioning

confidence: 99%

The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces

Jin,

Liu,

2021

Preprint

View full text Add to dashboard Cite

Modern reinforcement learning (RL) commonly engages practical problems with large state spaces, where function approximation must be deployed to approximate either the value function or the policy. While recent progresses in RL theory address a rich set of RL problems with general function approximation, such successes are mostly restricted to the single-agent setting. It remains elusive how to extend these results to multi-agent RL, especially in the face of new game-theoretical challenges. This paper considers two-player zero-sum Markov Games (MGs). We propose a new algorithm that can provably find the Nash equilibrium policy using a polynomial number of samples, for any MG with low multi-agent Bellman-Eluder dimension-a new complexity measure adapted from its single-agent version [26]. A key component of our new algorithm is the exploiter, which facilitates the learning of the main player by deliberately exploiting her weakness. Our theoretical framework is generic, which applies to a wide range of models including but not limited to tabular MGs, MGs with linear or kernel function approximation, and MGs with rich observations.

show abstract

Section: Introductionmentioning

confidence: 99%

The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces

Jin,

Liu,

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In the online setting, there is a recent line of research that proposes provably efficient RL algorithms for zero-sum Markov games. See, e.g., Wei et al (2017) Compared with these aforementioned works, we focus on solving the Stackelberg-Nash equilibrium, which involves a bilevel structure and is fundamentally different from the Nash equilibrium. Thus, our work is not directly comparable.…”

Section: Related Workmentioning

confidence: 99%

Can Reinforcement Learning Find Stackelberg-Nash Equilibria in General-Sum Markov Games with Myopic Followers?

Zhong¹,

Yang²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

We study multi-player general-sum Markov games with one of the players designated as the leader and the other players regarded as followers. In particular, we focus on the class of games where the followers are myopic, i.e., they aim to maximize their instantaneous rewards. For such a game, our goal is to find a Stackelberg-Nash equilibrium (SNE), which is a policy pair (π * , ν * ) such that (i) π * is the optimal policy for the leader when the followers always play their best response, and (ii) ν * is the best response policy of the followers, which is a Nash equilibrium of the followers' game induced by π * . We develop sample-efficient reinforcement learning (RL) algorithms for solving for an SNE in both online and offline settings. Our algorithms are optimistic and pessimistic variants of least-squares value iteration, and they are readily able to incorporate function approximation tools in the setting of large state spaces.Furthermore, for the case with linear function approximation, we prove that our algorithms achieve sublinear regret and suboptimality under online and offline setups respectively. To the best of our knowledge, we establish the first provably efficient RL algorithms for solving for SNEs in general-sum Markov games with myopic followers.

show abstract

“…Stochastic games (SGs) (Shapley 1953;Deng et al 2021) offer a multi-player game framework where agents jointly decide the loss and the state transition. Compared to OMDPs, the main difference is that SGs allow each player to have representation of states, actions and rewards, thus players can learn the representations over time and find the NE of the stochastic games (Wei, Hong, and Lu 2017;Tian et al 2020). The performance in SGs is often measured by the difference between the average loss and the value of the game (i.e., the value when both players play a NE), which is a weaker notion of regret compared to the best fixed policy in hindsight in OMDPs.…”

Section: Related Workmentioning

confidence: 99%

“…Intuitively, the player can learn the structure of the game (i.e., transition model, reward function) over time, thus on average, the player can calculate and compete with the value of the game. In non-episodic settings, the Upper Confidence Stochastic Game algorithm (UCSG) (Wei, Hong, and Lu 2017)…”

Section: Related Workmentioning

confidence: 99%

Online Markov Decision Processes with Non-oblivious Strategic Adversary

Dinh,

Mguni,

Tran-Thanh

et al. 2021

Preprint

View full text Add to dashboard Cite

We study a novel setting in Online Markov Decision Processes (OMDPs) where the loss function is chosen by a nonoblivious strategic adversary who follows a no-external regret algorithm. In this setting, we first demonstrate that MDP-Expert, an existing algorithm that works well with oblivious adversaries can still apply and achieve a policy regret bound of O ( log(where is the size of adversary's pure strategy set and | | denotes the size of agent's action space. Considering real-world games where the support size of a NE is small, we further propose a new algorithm: MDP-Online Oracle Expert (MDP-OOE), that achieves a policy regret bound of O ( log( ) + 2 log( )) where depends only on the support size of the NE. MDP-OOE leverages the key benefit of Double Oracle in game theory and thus can solve games with prohibitively large action space. Finally, to better understand the learning dynamics of no-regret methods, under the same setting of no-external regret adversary in OMDPs, we introduce an algorithm that achieves last-round convergence result to a NE. To our best knowledge, this is first work leading to the last iteration result in OMDPs.

show abstract

Online Reinforcement Learning in Stochastic Games

Cited by 17 publications

References 7 publications

The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces

The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces

Can Reinforcement Learning Find Stackelberg-Nash Equilibria in General-Sum Markov Games with Myopic Followers?

Online Markov Decision Processes with Non-oblivious Strategic Adversary

Contact Info

Product

Resources

About