Quantifying the Burden of Exploration and the Unfairness of Free Riding

Abstract

We consider the multi-armed bandit setting with a twist. Rather than having just one decision maker deciding which arm to pull in each round, we have n different decision makers (agents). In the simple stochastic setting, we show that a "free-riding" agent observing another "self-reliant" agent can achieve just O(1) regret, as opposed to the regret lower bound of Ω(log t) when one decision maker is playing in isolation. This result holds whenever the self-reliant agent's strategy satisfies either one of two assumptions: (1) each arm is pulled at least γ ln t times in expectation for a constant γ that we compute, or (2) the self-reliant agent achieves o(t) realized regret with high probability. Both of these assumptions are satisfied by standard zero-regret algorithms. Under the second assumption, we further show that the free rider only needs to observe the number of times each arm is pulled by the self-reliant agent, and not the rewards realized.In the linear contextual setting, each arm has a distribution over parameter vectors, each agent has a context vector, and the reward realized when an agent pulls an arm is the inner product of that agent's context vector with a parameter vector sampled from the pulled arm's distribution. We show that the free rider can achieve O(1) regret in this setting whenever the free rider's context is a small (in L 2 -norm) linear combination of other agents' contexts and all other agents pull each arm Ω(log t) times with high probability. Again, this condition on the self-reliant players is satisfied by standard zero-regret algorithms like UCB. We also prove a number of lower bounds.

Fair Exploration via Axiomatic Bargaining

“…This paper provides the first framework in evaluating fairness for allocating the burden of exploration in a group learning setting. Though the phenomenon motivating this problem has been previously observed (Jung et al 2020, Raghavan et al 2018, this work contains the first positive results to the best of our knowledge. Our work establishes a rigorous, axiomatic framework that can be readily applied to a plethora of different learning models.…”

“…Jung et al (2020) study a setting with multiple agents with a common bandit problem, where each agent can decide which action to…”

Multi-disciplinary fairness considerations in machine learning for clinical trials

“…There has been comparatively sparse research on this conception of unfairness, which seeks to examine the burdens placed on certain subgroups, evaluating fairness from the perspective of the agents or users that are affected by the model outcomes, rather than the arms/actions. Jung et al [127] demonstrate the problem of "free riding" in a multi-agent setting, where "free-riders" can incur minimal regret by accessing the information garnered by other agents. Raghavan et al [200] introduce the concept of group externalities to quantify the negative impacts the presence of one group may impose on another, under the linear contextual bandits model.…”