The Perils of Exploration under Competition

Aridor, Guy; Liu, Kevin; Slivkins, Aleksandrs; Wu, Zhiwei Steven

doi:10.1145/3328526.3329635

Cited by 10 publications

(10 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An algorithm may even enter a "death spiral", when the short-term reputation cost decreases the number of users for the algorithm to learn from, which degrades the system's performance relative to competition and further decreases the market share. Mansour et al (2018); Aridor et al (2019) investigate whether competition incentivizes the adoption of better bandit algorithms, and how it depends on the intensity of the competition. They relate these questions to the relationship between competition and innovation, a well-studied topic in economics (Schumpeter, 1942;Aghion et al, 2005).…”

Section: Agents Choose Between Bandit Algorithmsmentioning

confidence: 99%

Introduction to Multi-Armed Bandits

Slivkins¹

2019

Preprint

Self Cite

View full text Add to dashboard Cite

Multi-armed bandits a simple but very powerful framework for algorithms that make decisions over time under uncertainty. An enormous body of work has accumulated over the years, covered in several books and surveys. This book provides a more introductory, textbook-like treatment of the subject. Each chapter tackles a particular line of work, providing a self-contained, teachable technical introduction and a brief review of the further developments.The chapters are as follows:1. Stochastic bandits 2. Lower bounds 3. Bayesian Bandits and Thompson Sampling 4. Lipschitz Bandits 5. Full Feedback and Adversarial Costs 6. Adversarial Bandits 7. Linear Costs and Semi-bandits 8. Contextual Bandits 9. Bandits and Games 10. Bandits with Knapsacks 11. Bandits and Incentives Status of the manuscript: complete, but comments are very welcome! I plan to post revisions as I receive feedback and bug reports.To be published with Foundations and Trends in Machine Learning.

show abstract

Section: Agents Choose Between Bandit Algorithmsmentioning

confidence: 99%

Introduction to Multi-Armed Bandits

Slivkins¹

2019

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…It could well occur that the initial samples are less effective because, e.g., initially the advertised product is not yet well established which leads customers to prefer alternative options. In fact, recent work has suggested that, under competition, learning algorithms generally suffer from such an effect due to the exploration they need to perform in the beginning; see [MSW18] and [ALSW19] for a relevant discussion.…”

Section: Cold Start Attackmentioning

confidence: 99%

Bandits with adversarial scaling

Lykouris¹,

Mirrokni²,

Leme³

2020

Preprint

View full text Add to dashboard Cite

We study adversarial scaling, a multi-armed bandit model where rewards have a stochastic and an adversarial component. Our model captures display advertising where the click-throughrate can be decomposed to a (fixed across time) arm-quality component and a non-stochastic user-relevance component (fixed across arms). Despite the relative stochasticity of our model, we demonstrate two settings where most bandit algorithms suffer. On the positive side, we show that two algorithms, one from the action elimination and one from the mirror descent family are adaptive enough to be robust to adversarial scaling. Our results shed light on the robustness of adaptive parameter selection in stochastic bandits, which may be of independent interest.

show abstract

“…Aridor, Liu, Slivkins, and Wu [ALSW19] empirically study the interplay between exploration and competition in a model where multiple firms are competing for the same market of users and each firm commits to a multi-armed bandit algorithm. The objective of each firm is to maximize its market share and the question is when firms are incentivized to adopt better algorithms.…”

Section: Bolton and Harrismentioning

confidence: 99%

Multiplayer Bandit Learning, from Competition to Cooperation

Brânzei¹,

Peres²

2019

Preprint

View full text Add to dashboard Cite

The stochastic multi-armed bandit model captures the tradeoff between exploration and exploitation. We study the effects of competition and cooperation on this tradeoff. Suppose there are k arms and two players, Alice and Bob. In every round, each player pulls an arm, receives the resulting reward, and observes the choice of the other player but not their reward. Alice's utility is Γ A + λΓ B (and similarly for Bob), where Γ A is Alice's total reward and λ ∈ [−1, 1] is a cooperation parameter. At λ = −1 the players are competing in a zero-sum game, at λ = 1, they are fully cooperating, and at λ = 0, they are neutral: each player's utility is their own reward. The model is related to the economics literature on strategic experimentation, where usually players observe each other's rewards.With discount factor β, the Gittins index reduces the one-player problem to the comparison between a risky arm, with a prior µ, and a predictable arm, with success probability p. The value of p where the player is indifferent between the arms is the Gittins index g = g(µ, β) > m, where m is the mean of the risky arm.We show that competing players explore less than a single player: there is p * ∈ (m, g) so that for all p > p * , the players stay at the predictable arm. However, the players are not myopic: they still explore for some p > m. On the other hand, cooperating players explore more than a single player. We also show that neutral players learn from each other, receiving strictly higher total rewards than they would playing alone, for all p ∈ (p * , g), where p * is the threshold from the competing case.Finally, we show that competing and neutral players eventually settle on the same arm in every Nash equilibrium, while this can fail for cooperating players.

show abstract

The Perils of Exploration under Competition

Cited by 10 publications

References 55 publications

Introduction to Multi-Armed Bandits

Introduction to Multi-Armed Bandits

Bandits with adversarial scaling

Multiplayer Bandit Learning, from Competition to Cooperation

Contact Info

Product

Resources

About