DREAM: Deep Regret minimization with Advantage baselines and Model-free learning

Steinberger, Eric; Lerer, Adam; Brown, Noam

doi:10.48550/arxiv.2006.10410

Cited by 6 publications

(7 citation statements)

References 21 publications

(47 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There have been a number of RL algorithms that have been proposed for two-player zero-sum games: Fictitious Self-Play (47), Policy-Space Response Oracles (PSRO) (48), Double Neural CFR (49), Deep CFR and DREAM (50,51), Regret Policy Gradients (52), Exploitability Descent (53), Neural Replicator Dynamics (NeuRD) (54), Advantage Regret-Matching Actor Critic (55), Friction FoReL (56), Extensive-form Double Oracle (XDO) (57), Neural Auto-curricula (NAC) (58), and Regularized Nash Dynamics (R-NaD) (59). These methods adapt classical algorithms for computing (approximate) Nash equilibria to the RL setting with sampled experience and general function approximation.…”

Section: Related Workmentioning

confidence: 99%

Student of Games: A unified learning algorithm for both perfect and imperfect information games

Schmid,

Moravčík,

Burch

et al. 2023

Sci. Adv.

View full text Add to dashboard Cite

Games have a long history as benchmarks for progress in artificial intelligence. Approaches using search and learning produced strong performance across many perfect information games, and approaches using game-theoretic reasoning and learning demonstrated strong performance for specific imperfect information poker variants. We introduce Student of Games, a general-purpose algorithm that unifies previous approaches, combining guided search, self-play learning, and game-theoretic reasoning. Student of Games achieves strong empirical performance in large perfect and imperfect information games—an important step toward truly general algorithms for arbitrary environments. We prove that Student of Games is sound, converging to perfect play as available computation and approximation capacity increases. Student of Games reaches strong performance in chess and Go, beats the strongest openly available agent in heads-up no-limit Texas hold’em poker, and defeats the state-of-the-art agent in Scotland Yard, an imperfect information game that illustrates the value of guided search, learning, and game-theoretic reasoning.

show abstract

Section: Related Workmentioning

confidence: 99%

Student of Games: A unified learning algorithm for both perfect and imperfect information games

Schmid,

Moravčík,

Burch

et al. 2023

Sci. Adv.

View full text Add to dashboard Cite

show abstract

“…It is time-consuming for CFR to traverse the full game tree in solving large-scale imperfect-information extensive-form games. Therefore, to solve large-scale imperfect-information extensiveform games, some sampling-based CFR algorithms are proposed, such as external sampling, outcome sampling [8] , probe sampling [9] and other reduce variance sampling algorithms [22,23] . These sampling-based CFR algorithms only traverse a subset of the full game tree.…”

Section: U σ I (I) I I Imentioning

confidence: 99%

Decision Making in Team-Adversary Games with Combinatorial Action Space

Li,

Zhang,

Wang

et al. 2023

CAAI Artificial Intelligence Research

View full text Add to dashboard Cite

The team-adversary game simulates many real-world scenarios in which a team of agents competes cooperatively against an adversary. However, decision-making in this type of game is a big challenge since the joint action space of the team is combinatorial and exponentially related to the number of team members. It also hampers the existing equilibrium finding algorithms from solving team-adversary games efficiently. To solve this issue caused by the combinatorial action space, we propose a novel framework based on Counterfactual Regret Minimization (CFR) framework: CFR-MIX. Firstly, we propose a new strategy representation to replace the traditional joint action strategy by using the individual action strategies of all the team members, which can significantly reduce the strategy space. To maintain the cooperation between team members, a strategy consistency relationship is proposed. Then, we transform the consistency relationship of the strategy to the regret consistency for computing the equilibrium strategy with the new strategy representation under the CFR framework. To guarantee the regret consistency relationship, a product-form decomposition method over cumulative regret values is proposed. To implement this decomposition method, our CFR-MIX framework employs a mixing layer under the CFR framework to get the final decision strategy for the team, i.e., the Nash equilibrium strategy. Finally, we conduct experiments on games in different domains. Extensive results show that CFR-MIX significantly outperforms state-of-the-art algorithms. We hope it can help the team make decisions in large-scale teamadversary games.

show abstract

“…In this section, we use the same notation as in DREAM (Steinberger et al, 2020). An extensive-form game progresses through a sequence of player actions, and has a world state w P W at each step.…”

Section: Extensive-form Gamesmentioning

confidence: 99%

“…However, Deep CFR uses external sampling, which may be impractical for games with a large branching factor such as Stratego and Barrage Stratego. DREAM (Steinberger et al, 2020) and ARMAC (Gruslys et al, 2020) are model-free regret-based deep learning approaches.…”

Section: Related Workmentioning

confidence: 99%

XDO: A Double Oracle Algorithm for Extensive-Form Games

McAleer¹,

Lanier²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Policy Space Response Oracles (PSRO) is a deep reinforcement learning algorithm for two-player zero-sum games that has empirically found approximate Nash equilibria in large games. Although PSRO is guaranteed to converge to a Nash equilibrium, it may take an exponential number of iterations as the number of infostates grows. We propose Extensive-Form Double Oracle (XDO), an extensive-form double oracle algorithm that is guaranteed to converge to an approximate Nash equilibrium linearly in the number of infostates. Unlike PSRO, which mixes best responses at the root of the game, XDO mixes best responses at every infostate. We also introduce Neural XDO (NXDO), where the best response is learned through deep RL. In tabular experiments on Leduc poker, we find that XDO achieves an approximate Nash equilibrium in a number of iterations 1-2 orders of magnitude smaller than PSRO. In experiments on a modified Leduc poker game, we show that tabular XDO achieves over 11x lower exploitability than CFR and over 82x lower exploitability than PSRO and XFP in the same amount of time. We also show that NXDO beats PSRO and is competitive with NFSP on a large no-limit poker game.

show abstract

DREAM: Deep Regret minimization with Advantage baselines and Model-free learning

Cited by 6 publications

References 21 publications

Student of Games: A unified learning algorithm for both perfect and imperfect information games

Student of Games: A unified learning algorithm for both perfect and imperfect information games

Decision Making in Team-Adversary Games with Combinatorial Action Space

XDO: A Double Oracle Algorithm for Extensive-Form Games

Contact Info

Product

Resources

About