Policy Optimization for Markov Games: Unified Framework and Faster Convergence

Abstract

This paper studies policy optimization algorithms for multi-agent reinforcement learning. We begin by proposing an algorithm framework for two-player zero-sum Markov Games in the full-information setting, where each iteration consists of a policy update step at each state using a certain matrix game algorithm, and a value update step with a certain learning rate. This framework unifies many existing and new policy optimization algorithms. We show that the state-wise average policy of this algorithm converges to an approximate Nash equilibrium (NE) of the game, as long as the matrix game algorithms achieve low weighted regret at each state, with respect to weights determined by the speed of the value updates. Next, we show that this framework instantiated with the Optimistic Follow-The-Regularized-Leader (OFTRL) algorithm at each state (and smooth value updates) can find an O(T −5/6 ) approximate NE in T iterations, which improves over the current best O(T −1/2 ) rate of symmetric policy optimization type algorithms. We also extend this algorithm to multi-player general-sum Markov Games and show an O(T −3/4 ) convergence rate to Coarse Correlated Equilibria (CCE). Finally, we provide a numerical example to verify our theory and investigate the importance of smooth value updates, and find that using "eager" value updates instead (equivalent to the independent natural policy gradient algorithm) may significantly slow down the convergence, even on a simple game with H = 2 layers.

Can We Find Nash Equilibria at a Linear Rate in Markov Games?

“…A classical averaging stepsize from Jin et al (2018) is utilized by the critic so that the errors accumulate slowly and last-iterate convergence is obtained. Zhang et al (2022) propose a modified OFTRL method, where the min-player and the max-players employ a lower and upper bound for value functions separately. The lower and upper bounds are computed from approximate Q-functions in past iterations.…”

“…Global convergence. The proof for global convergence of Averaging OGDA adapts several standard techniques from Markov games Zhang et al (2022); Wei et al (2021). We attach its proof in Appendix C.1 for completeness.…”

“…However, due to the nonconvexity-nonconcavity, theoretical understanding of zero-sum Markov games is sparser. Existing methods have either sublinear rates for finding Nash equilibria, or linear rates for finding regularized Nash equiliria such as quantal response equilibria which are approximations for Nash equilibria Alacaoglu et al (2022); Cen et al (2021); Daskalakis et al (2020); Pattathil et al (2022); Perolat et al (2015); Wei et al (2021); Yang and Ma (2022); Zeng et al (2022); Zhang et al (2022); Zhao et al (2022). A natural question is:…”