Learning with stochastic inputs and adversarial outputs

Abstract

Most of the research in online learning is focused either on the problem of adversarial classification (i.e., both inputs and labels are arbitrarily chosen by an adversary) or on the traditional supervised learning problem in which samples are independent and identically distributed according to a stationary probability distribution. Nonetheless, in a number of domains the relationship between inputs and outputs may be adversarial, whereas input instances are i.i.d. from a stationary distribution (e.g., user preferences). This scenario can be formalized as a learning problem with stochastic inputs and adversarial outputs. In this paper, we introduce this novel stochastic-adversarial learning setting and we analyze its learnability. In particular, we show that in binary classification, given a hypothesis space H with finite VC-dimension, it is possible to design an algorithm which incrementally builds a suitable finite set of hypotheses from H used as input for an exponentially weighted forecaster and achieves a cumulative regret of order O( nV C(H) log n) with overwhelming probability. This result shows that whenever inputs are i.i.d. it is possible to solve any binary classification problem using a finite VCdimension hypothesis space with a sub-linear regret independently from the way labels are generated (either stochastic or adversarial). We also discuss extensions to multi-label classification, regression, learning from experts and bandit settings with stochastic side information, and application to games.

Online Markov Decision Processes Under Bandit Feedback

“…Thus, it remains to bound the expectation of the other two terms. This is done in the following two propositions whose proofs are deferred to the next subsections: 12 Lemma 5.2 of Even-Dar et al [7] gives a bound on νt − µ π t st 1 with a slightly different technique. However, there are multiple mistakes in the proof.…”

“…Finally, we note in passing that the contextual bandit problem considered by Lazaric and Munos [12] can also be regarded as a simplified version of our online learning problem where the states are generated in an i.i.d. fashion (though we do not consider the problem of competing with the best policy in a restricted subset of stationary policies).…”

Sequential prediction under log-loss with side information

“…In this scenario, we do not have access to another sequence X n . However, at time step i + 1, we have access to the past sequence X i which could be used, as done in [10], in lieu of X i . We now precisely define and analyze this probability assignment.…”

“…In particular, VCdim(G) < ∞ implies the PAClearnability of the hypothesis class G. Viewing the current setting as a log-loss variant of the standard classification problem studied in statistical learning (which uses the indicator loss) motivates the choice of constraint VCdim(G) < ∞. A variant of the current problem with indicator loss instead of log-loss was studied in [10]. We have considered a specific class of conditional distributions to compete against ( recall that under hypothesis f we have p f (Y = 0|X = x) = Bern(θ g(x) )).…”

Stochastic Structured Prediction under Bandit Feedback

“…Since we aim to train these models in interaction with real users, we focus on the ease of elicitability of the feedback and on speed of convergence. In the spectrum of stochastic versus adversarial bandits, our approach is semi-adversarial in making stochastic assumptions on inputs, but not on rewards [12].…”