Good arm identification via bandit feedback

Abstract

We consider a novel stochastic multi-armed bandit problem called good arm identification (GAI), where a good arm is defined as an arm with expected reward greater than or equal to a given threshold. GAI is a pure-exploration problem that a single agent repeats a process of outputting an arm as soon as it is identified as a good one before confirming the other arms are actually not good. The objective of GAI is to minimize the number of samples for each process. We find that GAI faces a new kind of dilemma, the exploration-exploitation dilemma of confidence, which is different difficulty from the best arm identification. As a result, an efficient design of algorithms for GAI is quite different from that for the best arm identification. We derive a lower bound on the sample complexity of GAI that is tight up to the logarithmic factor O(log 1 δ ) for acceptance error rate δ. We also develop an algorithm whose sample complexity almost matches the lower bound. We also confirm experimentally that our proposed algorithm outperforms naive algorithms in synthetic settings based on a conventional bandit problem and clinical trial researches for rheumatoid arthritis.

A bad arm existence checking problem: How to utilize asymmetric problem structure?

“…The algorithm using our stopping condition stopped drawing an arm about two times faster than the algorithm using the conventional stopping condition when its loss mean is around the center of the thresholds. Our algorithm with arm selection policy APT P always stopped faster than the algorithm using arm selection policy UCB (Auer and Cesa-Bianchi, 2002) like HDoC (Kano et al, 2017), and our algorithm's stopping time was faster or comparable to the stopping time of the algorithm using arm selection policy LUCB (Kalyanakrishnan et al, 2012) in our simulations using Bernoulli loss distribution with synthetically generated means and means generated from a real-world dataset.…”

“…Remark 1 Identification is not needed for checking existence, however, in terms of asymptotic behavior as δ → +0, the shown expected sample complexity lower bounds of both the tasks are the same; lim δ→+0 E(T )/ ln(1/δ) ≥ 1/d(µ 1 , θ L ) for both the tasks in the case with some positive arms. The bounds are tight considering the shown upper bounds, so the bad arm existence checking is not more difficult than the good arm identification (Kano et al, 2017) with respect to asymptotic behavior as δ → +0.…”

“…HDoC (Hybrid algorithm for the Dilemma of Confidence) (Kano et al, 2017) for good arm identification problem uses arm selection policy UCB (Upper Confidence Bound) (Auer and Cesa-Bianchi, 2002), in which…”

“…compared to the case using the conventional stopping condition of the successive elimination algorithm (Even-Dar et al, 2006). Regarding the asymptotic behavior as δ → 0, the upper bound on the expected number of samples for our algorithm with arm selection policy APT P is proved to be almost optimal when all the positive arms have the same loss mean, which is the case that HDoC (Kano et al, 2017) does not perform well. Note that HDoC is an algorithm for good arm identification that uses UCB (Auer and Cesa-Bianchi, 2002) as the arm selection policy.…”

“…Apart from whether fixed confidence (constraint on error probability to achieve) or fixed budget (constraint on the allowable number of draws), positive and negative arms are treated symmetrically in the thresholding bandit problem while they are dealt with asymmetrically in our problem setting; judgment of one positive arm existence is enough for positive conclusion though all the arms must be judged as negative for negative conclusion. This asymmetry has also been considered in the good arm identification problem (Kano et al, 2017), and our problem can be seen as its specialized version. In their setting, the player's task is to output all the arms of above-threshold means with probability at least 1 − δ, and his/her objective is to minimize the number of drawn samples until λ arms are outputted as arms with above-threshold means for a given λ.…”