Online Convex Optimization for Sequential Decision Processes and Extensive-Form Games

Abstract

Regret minimization is a powerful tool for solving large-scale extensive-form games. State-of-the-art methods rely on minimizing regret locally at each decision point. In this work we derive a new framework for regret minimization on sequential decision problems and extensive-form games with general compact convex sets at each decision point and general convex losses, as opposed to prior work which has been for simplex decision points and linear losses. We call our framework laminar regret decomposition. It generalizes the CFR algorithm to this more general setting. Furthermore, our framework enables a new proof of CFR even in the known setting, which is derived from a perspective of decomposing polytope regret, thereby leading to an arguably simpler interpretation of the algorithm. Our generalization to convex compact sets and convex losses allows us to develop new algorithms for several problems: regularized sequential decision making, regularized Nash equilibria in extensive-form games, and computing approximate extensive-form perfect equilibria. Our generalization also leads to the first regret-minimization algorithm for computing reduced-normal-form quantal response equilibria based on minimizing local regrets. Experiments show that our framework leads to algorithms that scale at a rate comparable to the fastest variants of counterfactual regret minimization for computing Nash equilibrium, and therefore our approach leads to the first algorithm for computing quantal response equilibria in extremely large games. Finally we show that our framework enables a new kind of scalable opponent exploitation approach.

Model-Free Learning for Two-Player Zero-Sum Partially Observable Markov Games with Perfect Recall

“…Our choice of regularizer allows an efficient update of the current policy with a O(HA) time-complexity per episode (see Section 3.3). In particular, our result answers the open problem raised by Farina et al (2021b) and Farina and Sandholm (2021) of providing an algorithm with high-probability regret bound scaling with √ T with O(HA) computations per episode. Interestingly, we can also update the average profile (which will be returned at the end of the learning, see Section 3.3) in an online fashion.…”

“…Note that this bound is a consequence of a bound on the regret of both players (see Section 2) that holds even in the non-stochastic setting where an adversary picks a new game at each episode. Closer to our approach, Farina et al (2021b) recast the setting to an adversarial bandit linear optimization (Flaxman et al, 2005;Abernethy et al, 2008, see also Section 3.1). Precisely, they use the online mirror descent (OMD) algorithm with the dilated entropy distance-generating function (Hoda et al, 2010;Kroer et al, 2015) as regularizer.…”

Solving Imperfect-Information Games via Discounted Regret Minimization

“…Although CFR theory calls for both players to simultaneously update their regrets on each iteration, in practice far better performance is achieved by alternating which player updates their regrets on each iteration. However, this complicates the theory for convergence (Farina, Kroer, and Sandholm ;Burch, Moravcik, and Schmid 2018). CFR+ is like CFR but with the following small changes.…”

An Adaptive Similarity-Measuring-Based CMAB Model for Recommendation System

“…Sequential decision making [1] has received much attention in recent years due to the rapid increase in the generation of the streaming data. One representative application field of decision making problems is recommendation system [2], where an agent has to make a decision from several choices, e.g., assign the users with appropriate advertisements [3], articles [4], movies [5] and etc [6].…”