Learning to Detect an Odd Markov Arm

Abstract

A multi-armed bandit with finitely many arms is studied when each arm is a homogeneous Markov process on an underlying finite state space. The transition law of one of the arms, referred to as the odd arm, is different from the common transition law of all other arms. A learner, who has no knowledge of the above transition laws, has to devise a sequential test to identify the index of the odd arm as quickly as possible, subject to an upper bound on the probability of error. For this problem, we derive an asymptotic lower bound on the expected stopping time of any sequential test of the learner, where the asymptotics is as the probability of error vanishes. Furthermore, we propose a sequential test, and show that the asymptotic behaviour of its expected stopping time comes arbitrarily close to that of the lower bound. Prior works deal with independent and identically distributed arms, whereas our work deals with Markov arms. Our analysis of the rested Markov setting is a key first step in understanding the difficult case of restless Markov setting, which is still open. Index TermsMulti-armed bandits, rested bandits, Markov rewards, odd arm identification, anomaly detection, forced exploration.

Detecting an Odd Restless Markov Arm with a Trembling Hand

“…Both the assumptions were crucial in establishing that the ML estimates of the TPMs converge to their true values. In [7]- [9], an analogue of the continuous selection property was established for the maximisers instead of δ-optimal solutions.…”

“…It is worth noting here that in the prior works [7]- [9], the outer supremum in the expression for the constant appearing in the lower bound is over all unconditional probability distributions on the arms which are simpler objects to deal with than conditional probability distributions. Further, in these works, this outer supremum is attained by a unique (unconditional) probability distribution on the arms.…”

“…4328], the final expression for the lower bound when the arms are rested. The absence of arm delays in [9,Eq. (19), pp.…”

“…Extension of the results in [7] to the case when each arm yields iid observations coming from a generic exponential family appears in [8]. The setting when each arm yields Markov observations and the arms are rested has been studied in [9]. The recent papers [10]- [12] study more general sequential hypothesis testing problems in multi-armed bandits of which the problem of odd arm identification is a special case.…”

“…3) An important property of the optimal solutions to the lower bounds in [7]- [9] is that they are continuous functions of the arms configuration. It is this continuity property that is used to show that the certainty equivalence-based arm sampling policies of these works (using ML estimation) render the system identifiable.…”