A Counter-example to Karlin's Strong Conjecture for Fictitious Play

Daskalakis, Constantinos; Pan, Qinxuan

doi:10.1109/focs.2014.10

Cited by 17 publications

(14 citation statements)

References 32 publications

(44 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Within this line of work, perhaps the most prominent dynamic is the "fictitious play" algorithm, in which both players repeatedly follow their best-response to the empirical distribution of their opponent's past plays. This simple and natural dynamic was first proposed by Brown (1951), shown to converge to equilibrium in twoplayer zero-sum games by Robinson (1951), and was extensively studied ever since (see e.g., Brandt et al, 2013;Daskalakis and Pan, 2014 and the references therein). Another related dynamic, put forth by Hannan (1957) and popularized by Kalai and Vempala (2005), is based on perturbed (i.e., noisy) best-responses.…”

Section: Related Workmentioning

confidence: 99%

The computational power of optimization in online learning

Hazan

Koren

2016

Proceedings of the Forty-Eighth Annual ACM Symposium on Theory of Computing

View full text Add to dashboard Cite

We consider the fundamental problem of prediction with expert advice where the experts are "optimizable": there is a black-box optimization oracle that can be used to compute, in constant time, the leading expert in retrospect at any point in time. In this setting, we give a novel online algorithm that attains vanishing regret with respect to N experts in total O( √ N ) computation time. We also give a lower bound showing that this running time cannot be improved (up to log factors) in the oracle model, thereby exhibiting a quadratic speedup as compared to the standard, oracle-free setting where the required time for vanishing regret is Θ(N ). These results demonstrate an exponential gap between the power of optimization in online learning and its power in statistical learning: in the latter, an optimization oracle-i.e., an efficient empirical risk minimizer-allows to learn a finite hypothesis class of size N in time O(log N ).We also study the implications of our results to learning in repeated zero-sum games, in a setting where the players have access to oracles that compute, in constant time, their best-response to any mixed strategy of their opponent. We show that the runtime required for approximating the minimax value of the game in this setting is Θ( √ N ), yielding again a quadratic improvement upon the oracle-free setting, where Θ(N ) is known to be tight.

show abstract

Section: Related Workmentioning

confidence: 99%

The computational power of optimization in online learning

Hazan

Koren

2016

Proceedings of the Forty-Eighth Annual ACM Symposium on Theory of Computing

View full text Add to dashboard Cite

show abstract

“…However, simple games such as Rock-Paper-Scissors illustrate that such a greedy approach is not guaranteed to converge to a stationary point. A slight variant, fictitious play [Brown, 1951] does converge to a Nash equilibrium in finite time Daskalakis and Pan [2014], Robinson [1951]. At each iteration, each player chooses their best strategy in response to the historical average of the opponent's strategies.…”

Section: Optimizing the State Marginal Matching Objectivementioning

confidence: 99%

Efficient Exploration via State Marginal Matching

Lee,

Eysenbach,

Parisotto

et al. 2019

Preprint

View full text Add to dashboard Cite

To solve tasks with sparse rewards, reinforcement learning algorithms must be equipped with suitable exploration techniques. However, it is unclear what underlying objective is being optimized by existing exploration algorithms, or how they can be altered to incorporate prior knowledge about the task. Most importantly, it is difficult to use exploration experience from one task to acquire exploration strategies for another task. We address these shortcomings by learning a single exploration policy that can quickly solve a suite of downstream tasks in a multi-task setting, amortizing the cost of learning to explore. We recast exploration as a problem of State Marginal Matching (SMM): we learn a mixture of policies for which the state marginal distribution matches a given target state distribution, which can incorporate prior knowledge about the task. Without any prior knowledge, the SMM objective reduces to maximizing the marginal state entropy. We optimize the objective by reducing it to a two-player, zero-sum game, where we iteratively fit a state density model and then update the policy to visit states with low density under this model. While many previous algorithms for exploration employ a similar procedure, they omit a crucial historical averaging step, without which the iterative procedure does not converge to a Nash equilibria. To parallelize exploration, we extend our algorithm to use mixtures of policies, wherein we discover connections between SMM and previously-proposed skill learning methods based on mutual information. On complex navigation and manipulation tasks, we demonstrate that our algorithm explores faster and adapts more quickly to new tasks. 1

show abstract

“…Improving on Shapiro's upper bound, Karlin's strong conjecture says (or more precisely, said) that DTFP converges in payo¤s at rate O(t 1=2 ) in two-person zero-sum games, regardless of the number of strategies. Daskalakis and Pan (2014) have recently disproved that conjecture, showing that the rate of convergence in an asymmetric two-person zero-sum game in which both players have the same number of strategies may be as low as O(t 1= ). That lower bound holds, obviously, also for zero-sum networks.…”

Section: =1mentioning

confidence: 99%

Fictitious play in networks

Ewerhart

Valkanova

2020

Games and Economic Behavior

View full text Add to dashboard Cite

This paper studies …ctitious play in networks of noncooperative twoperson games. We show that continuous-time …ctitious play converges to the set of Nash equilibria if the overall n-person game is zero-sum. Moreover, the rate of convergence is 1= , regardless of the size of the network. In contrast, arbitrary nperson zero-sum games with bilinear payo¤ functions do not possess the continuoustime …ctitious-play property. As extensions, we consider networks in which each bilateral game is either strategically zero-sum, a weighted potential game, or a twoby-two game. In those cases, convergence requires a condition on bilateral payo¤s or, alternatively, that the network is acyclic. Our results hold also for the discrete-time variant of …ctitious play, which implies, in particular, a generalization of Robinson's theorem to arbitrary zero-sum networks. Applications include security games, con ‡ict networks, and decentralized wireless channel selection.

show abstract

A Counter-example to Karlin's Strong Conjecture for Fictitious Play

Abstract: Fictitious play is a natural dynamic for equilibrium play in zero-sum games, proposed by 2

Cited by 17 publications

References 32 publications

The computational power of optimization in online learning

The computational power of optimization in online learning

Efficient Exploration via State Marginal Matching

Fictitious play in networks

Contact Info

Product

Resources

About