Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibrium

Abstract

We develop provably efficient reinforcement learning algorithms for two-player zero-sum Markov games in which the two players simultaneously take actions. To incorporate function approximation, we consider a family of Markov games where the reward function and transition kernel possess a linear structure. Both the offline and online settings of the problems are considered. In the offline setting, we control both players and the goal is to find the Nash Equilibrium efficiently by minimizing the worst-case duality gap. In the online setting, we control a single player to play against an arbitrary opponent and the goal is to minimize the regret. For both settings, we propose an optimistic variant of the least-squares minimax value iteration algorithm. We show that our algorithm is computationally efficient and provably achieves an O( √ d 3 H 3 T) upper bound on the duality gap and regret, without requiring additional assumptions on the sampling model.We highlight that our setting requires overcoming several new challenges that are absent in Markov decision processes or turn-based Markov games. In particular, to achieve optimism in simultaneous-move Marko games, we construct both upper and lower confidence bounds of the value function, and then compute the optimistic policy by solving a general-sum matrix game with these bounds as the payoff matrices. As finding the Nash Equilibrium of such a general-sum game is computationally hard, our algorithm instead solves for a Coarse Correlated Equilibrium (CCE), which can be obtained efficiently via linear programming. To our best knowledge, such a CCE-based scheme for implementing optimism has not appeared in the literature and might be of interest in its own right.

Towards General Function Approximation in Zero-Sum Markov Games

“…We propose an algorithm that incurs at most O(H √ d E K log N F ) 1 regret in K episodes where d E denotes the Minimax Eluder dimension N F denotes the covering number of function class. As a special case, this result improves [XCWY20] by a √ d multiplicative factor when the reward function and transition kernel are linearly parameterized and d is the dimension of feature mapping.…”

“…However, without strong sampling model or a well explored policy, the issue of exploration-exploitation tradeoff must addressed. Most of these work focus on tabular setting [WHL17, PSPP17, BJ20, BJWX21, BJY20, ZKBY20, ZTLD21] or linear function approximation settings [XCWY20]. Moreover, the concurrent work of [JLY21] studies competitive RL with general function approximation.…”

V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RL

“…While several provable decentralized MARL algorithms have been developed [see, e.g., 57,40,13], they either have only asymptotic guarantees or work only under certain reachability assumptions (see Section 1.1). The existing provably efficient algorithms for general Markov games (without further assumptions) are exclusively centralized algorithms [2,55,30].…”

“…In this section, we focus our attention on theoretical results for the tabular setting, where the numbers of states and actions are finite. We acknowledge that there has been much recent work in RL for continuous state spaces [see, e.g., 21,23,56,24,55,25], but this setting is beyond our scope.…”

“…A recent line of works provide non-asymptotic guarantees for learning two-player zero-sum tabular MGs without further structural assumptions. Bai and Jin [2] and Xie et al [55] develop the first provably-efficient learning algorithms in MGs based on optimistic value iteration. Liu et al [30] improves upon these works and achieve best-known sample complexity for finding an ǫ-Nash equilibrium-O(H 3 SA 1 A 2 /ǫ 2 ) episodes.…”