Using Response Functions to Measure Strategy Strength

Abstract

Extensive-form games are a powerful tool for representing complex multi-agent interactions. Nash equilibrium strategies are commonly used as a solution concept for extensive-form games, but many games are too large for the computation of Nash equilibria to be tractable. In these large games, exploitability has traditionally been used to measure deviation from Nash equilibrium, and thus strategies are aimed to achieve minimal exploitability. However, while exploitability measures a strategy's worst-case performance, it fails to capture how likely that worst-case is to be observed in practice. In fact, empirical evidence has shown that a less exploitable strategy can perform worse than a more exploitable strategy in one-on-one play against a variety of opponents. In this work, we propose a class of response functions that can be used to measure the strength of a strategy. We prove that standard no-regret algorithms can be used to learn optimal strategies for a scenario where the opponent uses one of these response functions. We demonstrate the effectiveness of this technique in Leduc Hold'em against opponents that use the UCT Monte Carlo tree search algorithm.

Complexity and Algorithms for Exploiting Quantal Opponents in Large Two-Player Games

“…Because QSE and QNE are usually non-equivalent concepts even in zero-sum games (see Figure 1), the regretminimization algorithms will not converge to QSE. However, in case a quantal function satisfies the so-called prettygood-response condition, the algorithm converges to a strategy of the leader exploiting the follower the most (Davis, Burch, and Bowling 2014). We show that a class of simple (i.e., attaining only a finite number of values) quantal functions satisfy a pretty-good-responses condition.…”

“…However, when creating AI agents competing with humans, we want to assume that one of the players is perfectly rational, and only the opponent's rationality is bounded. A tempting approach may be using the algorithms for computing QRE and increasing one player's rationality or using generic algorithms for exploiting opponents (Davis, Burch, and Bowling 2014) even though the QR model does not satisfy their assumptions, as in (Basak et al 2018). However, this approach generally leads to a solution concept we call Quantal Nash Equilibrium (QNE), which we show is very inefficient in exploiting QR opponents and may even perform worse than an arbitrary Nash equilibrium.…”

“…Therefore, 3) we investigate the performance of CFR-based heuristics against QR opponents. The extensive empirical evaluation on four different classes of games with up to 10 8 histories identifies a variant of CFR-f (Davis, Burch, and Bowling 2014) that computes strategies better than both QNE and NE.…”

Estimating Agent Skill in Continuous Action Domains

“…Opponent modeling is the problem of estimating the properties of an opponent (Nashed & Zilberstein, 2022). Much previous work on this topic has been done in imperfect information games like poker (Billings et al, 1998;Bard et al, 2015Bard et al, , 2013Davis et al, 2014), but this work focuses on strategic characteristics and limitations of the opponents, and the domains do not include execution uncertainty. Examples of additional areas where opponent modeling work has been explored include general multi-agent systems (Carmel & Markovitch, 1995), real-time strategy games (Schadd et al, 2007), and n-player games (Sturtevant et al, 2006).…”

AlphaZe∗∗: AlphaZero-like baselines for imperfect information games are surprisingly strong

“…. Elo as metric Davis et al (2014) showed that in many large imperfect information games the computation of a Nash equilibrium is not tractable and measuring the deviation from it is not a good measurement for the quality of an agent in all cases, e.g., they showed that a more exploitable agent is able to beat a less exploitable agent in some situations. Furthermore, they argue that calculating the exploitability can become a problem in large games.…”