Model Selection in Contextual Stochastic Bandit Problems

Abstract

We study model selection in stochastic bandit problems. Our approach relies on a master algorithm that selects its actions among candidate base algorithms. While this problem is studied for specific classes of stochastic base algorithms, our objective is to provide a method that can work with more general classes of stochastic base algorithms. We propose a master algorithm inspired by CORRAL Agarwal et al. ( 2017) and introduce a novel and generic smoothing transformation for stochastic bandit algorithms that permits us to obtain O( √ T ) regret guarantees for a wide class of base algorithms when working along with our master. We exhibit a lower bound showing that even when one of the base algorithms has O(log T ) regret, in general it is impossible to get better than Ω( √ T ) regret in model selection, even asymptotically. We apply our algorithm to choose among different values of for the -greedy algorithm, and to choose between the k-armed UCB and linear UCB algorithms. Our empirical studies further confirm the effectiveness of our model-selection method.

Universal and data-adaptive algorithms for model selection in linear contextual bandits

“…Existing approaches to address either the stronger Objective 1 [9] or the weaker Objective 2 [16] make restrictive assumptions 2 regarding the conditioning (what we will call diversity) of the contexts. Other, more dataagnostic approaches [3,32,31,28] achieve neither of the above objectives. This leads us to ask whether we can design a universal model selection approach that is data-agnostic (other than requiring a probability model on the contexts) and achieves either Objective 1 or 2.…”

Misspecified Gaussian Process Bandit Optimization

“…[14] introduce a new family of algorithms that require access to an online oracle for square loss regression and address the case of adversarial contexts. Concurrent work of [33] solves the case when contexts / action sets are stochastic. Both works ( [14] and [33]) leverage CORRAL-type aggregation [2] of contextual bandit algorithms and achieve the optimal Õ( √ dT ǫ + d √ T ) regret bound.…”

“…Concurrent work of [33] solves the case when contexts / action sets are stochastic. Both works ( [14] and [33]) leverage CORRAL-type aggregation [2] of contextual bandit algorithms and achieve the optimal Õ( √ dT ǫ + d √ T ) regret bound. Finally, in [32], the authors present a practical master algorithm that plays base algorithms that come with a candidate regret bound that may not hold during all rounds.…”

“…The master algorithm plays base algorithms in a balanced way and suitably eliminates algorithms whose regret bound is no longer valid. Similarly to the previous works that rely on the CORRAL-type master algorithms, we use the balancing master algorithm of [33] together with our GP-bandit base algorithm to provide contextual misspecified regret bounds.…”

“…• For when ǫ is unknown, we propose another algorithm based on uncertainty sampling and phased exploration that achieves (up to polylog factors) the previous regret rates in the misspecified setting, and standard regret guarantees in the realizable case (when ǫ = 0). • Finally, we consider a misspecified contextual kernelized problem, and show that when action sets are stochastic, our EC-GP-UCB algorithm can be effectively combined with the regret bound balancing strategy from [33] to achieve previous regret bounds (up to some additive lower order terms).…”