Addressing the Long-term Impact of ML Decisions via Policy Regret

Abstract

Machine Learning (ML) increasingly informs the allocation of opportunities to individuals and communities in areas such as lending, education, employment, and beyond. Such decisions often impact their subjects' future characteristics and capabilities in an a priori unknown fashion. The decision-maker, therefore, faces exploration-exploitation dilemmas akin to those in multi-armed bandits. Following prior work, we model communities as arms. To capture the long-term effects of ML-based allocation decisions, we study a setting in which the reward from each arm evolves every time the decision-maker pulls that arm. We focus on reward functions that are initially increasing in the number of pulls but may become (and remain) decreasing after a certain point. We argue that an acceptable sequential allocation of opportunities must take an arm's potential for growth into account. We capture these considerations through the notion of policy regret, a much stronger notion than the often-studied external regret, and present an algorithm with provably sub-linear policy regret for sufficiently long time horizons. We empirically compare our algorithm with several baselines and find that it consistently outperforms them, in particular for long time horizons.

Complete Policy Regret Bounds for Tallying Bandits

“…Thus far, our motivation for the tallying bandit setting has been primarily theoretical, to resolve the gap in our understanding of when we can efficiently minimize CPR. Nevertheless, in similar vein to Heidari et al [HKR16], Lindner et al [LHK21] and Awasthi et al [ABGK22], we believe that the tallying bandit is a simple approximation for various practical settings. For instance, in recommender systems the reward associated with an action is rarely static, because the stimulus of recommended content influences user preferences [CLA + 03, SGR16].…”

“…A natural restriction is to enforce that each g x has special structure. This is precisely the approach taken by works on rotting bandits [HKR16, LCM17, SLC + 19, SMLV20], improving bandits [HKR16], single peaked bandits [LHK21] and congested bandits [ABGK22]. Concretely, these works use base functions {g x } x∈X that have the following special "tallying" structure.…”

“…Improving & Single Peaked Bandits. The improving bandit [HKR16] and single peaked bandit [LHK21] are both special cases of our tallying bandit setting, and deserve special attention. In improving bandits, the reward of an arm is an increasing, concave function of the number of times it has been pulled.…”

“…The strongest possible version of policy regret is when C T is the complete policy class (i.e., the set of all deterministic policies), and we abbreviate this as the complete policy regret. This challenging setting has recently received attention from the online learning community [HKR16,SMLV20,LHK21]. On the other hand, this performance metric is equivalent to the one that is standard in the closely related field of reinforcement learning, where a vast literature explores how to efficiently maximize cumulative reward [KMN99, SMSM00, KKL03, JAZBJ18, WDYK20, MPSL21].…”

“…We believe this is a worthwhile endeavor, since attaining sublinear CPR is a nontrivial task riddled with subtleties, even in settings that make much stronger assumptions than we do. For instance, the improving [HKR16] and single peaked [LHK21] bandit settings enforce m = T , require monotonicity and convexity conditions on {h x } x∈X , and also require the losses are observed deterministically. Even with these strong requirements, the best known CPR guarantees are asymptotic bounds that may decay arbitrarily slowly, and their algorithms cannot handle m < T .…”