Learning Zero-sum Stochastic Games with Posterior Sampling

Abstract

In this paper, we propose Posterior Sampling Reinforcement Learning for Zero-sum Stochastic Games (PSRL-ZSG), the first online learning algorithm that achieves Bayesian regret bound of O(HS √ AT ) in the infinite-horizon zero-sum stochastic games with average-reward criterion. Here H is an upper bound on the span of the bias function, S is the number of states, A is the number of joint actions and T is the horizon. We consider the online setting where the opponent can not be controlled and can take any arbitrary timeadaptive history-dependent strategy. This improves the best existing regret bound of O( 3 √ DS 2 AT 2 ) by (Wei, Hong, and Lu 2017) under the same assumption and matches the theoretical lower bound in A and T .

Decentralized model-free reinforcement learning in stochastic games with average-reward objective

“…This result is similar to the bound obtained in [24] for the regret in infinite-horizon MDP. In addition, our algorithm enjoys a low computational complexity and low memory space requirement compared to the previous works of [23] and [9] in the same setting. We also provide a high-probability bound for the regret of our algorithm.…”

“…This definition is classical for the regret in decentralized learning [9,21,23], and generalizes the definition for the MDP setting [1,24]. Note that in contrast with MDPs, the regret is not necessarily non-negative: If the opponent is weak, then the learning agent can achieve an average-reward greater than * .…”

“…In this paper we assume the same diameter assumption. In a recent work, inspired from PSRL algorithm for MDP, PSRL-ZSG algorithm is proposed in [9] and achieves a Bayesian regret of order √ under the same structure assumption. Our work does not improve the above regret bounds but the model-free property of our algorithm induces a lower computational complexity and memory space requirement than the previous model-based algorithms.…”

“…An interesting property of Q-learning like algorithms is that only the values of the current state and state-action tuple are updated at each step. This contrasts with model-based approaches [9,23] where some policies have to be entirely computed in the inner loop of the algorithm, or with value-iteration like algorithms [2,26] where a sweep of the state space is needed at each step. The only non-constant time operation in DONQ-learning is the call to Get-NashPolicies procedure, but it is known that we can compute the value and the Nash policy of a zero-sum matrix game of size × in a polynomial time via linear programming [8,11], and this does depend on the size of the state space.…”

“…In a model-free algorithm, the learner directly tries to estimate the values of the true model and plays optimally according to these value estimations. The two papers that are closest to ours are the algorithms UCSG in [23] and PSRL-ZSG in [9]. They both present a decentralized learning algorithm in averagereward SG and are both model-based.…”