Regression Oracles and Exploration Strategies for Short-Horizon Multi-Armed Bandits

Abstract

This paper explores multi-armed bandit (MAB) strategies in very short horizon scenarios, i.e., when the bandit strategy is only allowed very few interactions with the environment. This is an understudied setting in the MAB literature with many applications in the context of games, such as player modeling. Specifically, we pursue three different ideas. First, we explore the use of regression oracles, which replace the simple average used in strategies such as-greedy with linear regression models. Second, we examine different exploration patterns such as forced exploration phases. Finally, we introduce a new variant of the UCB1 strategy called UCBT that has interesting properties and no tunable parameters. We present experimental results in a domain motivated by exergames, where the goal is to maximize a player's daily steps. Our results show that the combination of-greedy or-decreasing with regression oracles outperforms all other tested strategies in the short horizon setting.

The Personalization Paradox: the Conflict between Accurate User Models and Personalized Adaptive Systems

“…The user's steps are captured by Fitbit and synced automatically with our platform. We use the AI technique of multi-armed bandits to model individual users' social comparison preferences and adapt the comparison targets shown to them [12,13]. For example, if the user model predicts that a particular user tends to prefer upward comparisons, the system will show more profiles with a larger number of daily steps.…”