Multiclass classification with bandit feedback using adaptive regularization

Abstract

We present a new multiclass algorithm in the bandit framework, where after making a prediction, the learning algorithm receives only partial feedback, i.e., a single bit of right-orwrong, rather then the true label. Our algorithm is based on the 2nd-order Perceptron, and uses upper-confidence bounds to trade off exploration and exploitation. We analyze this algorithm in a partial adversarial setting, where instances are chosen adversarially, while the labels are chosen according to a linear probabilistic model, which is also chosen adversarially. We show a regret of O( √ T log T ), which improves over the current best bounds of O(T 2/3 ) in the fully adversarial setting. We evaluate our algorithm on nine real-world text classification problems, obtaining state-of-the-art results, even compared with non-bandit online algorithms, especially when label noise is introduced.

Incremental learning using partial feedback for gesture-based human-swarm interaction

“…In this context, learning strategies to automatically adjust the trade-off parameter for the Bandit model were presented [13]. In [14], a multi-class classification scheme was introduced with the bandit feedback using adaptive regularization. The approach is based on the 2 nd -order perceptron and makes use of the upper confidence bounds (UCBs) [15] to balance the trade-off between exploration and exploitation for improving the bounds of the Banditron [10].…”

“…On the other hand RLS [16] methods based on linear regression has shown performance better than the Banditron class of algorithms [14] and similar to that of SVMs and other kernelbased methods [16], with the additional ability to provide confidence estimates of the predicted samples. Therefore, in this paper we propose interactive incremental learning algorithms derived from RLS methods.…”

“…However, when the feedback is an "incorrect prediction" (i.e., l t = −1), we can only infer just the binary label z i t of the chosen predictor i. Thus, the update rule is modified accordingly, as described in Algorithm 2 (lines [14][15][16][17][18][19][20][21][22]. Since each predictor may be updated differently, besides separate vectors b i as in the full feedback case, we need to maintain a separate matrix A i for each predictor i.…”

“…We adopt the widely used upper confidence bound UCB = γ i t + σ i t [15], [14] to optimistically select the predictor (line 12 of Algorithm 2).…”

“…//Predict with confidence interval It is worth mentioning that, similar to the approach employed in [14], our Algorithm 2 also maintains additional confidence information associated with each prediction, making an optimistic prediction using the highest UCB. However, as opposed to the algorithm in [14], our algorithm maintains separate binary predictors in a one-vs-all approach, allowing all the predictors to get updated when a positive feedback is available (lines 14-18 in Algorithm 2), being in this way more label-efficient.…”