Piecewise-stationary bandit problems with side observations

Yu, Jia Yuan; Mannor, Shie

doi:10.1145/1553374.1553524

Cited by 65 publications

(55 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Non-stationary environments have not been extensively studied in the bandit literature. For unstructured problems, the performance of algorithms based on UCB [3] has been analyzed in [12,22,37] under the assumption that the average rewards are abruptely changing. Here we consider more realistic scenarios where the average rewards smoothly evolve over time.…”

Section: Stochastic Mab Problemsmentioning

confidence: 99%

Optimal Rate Sampling in 802.11 systems

Combes

Proutière

Yun

et al. 2014

IEEE INFOCOM 2014 - IEEE Conference on Computer Communications

View full text Add to dashboard Cite

Rate Adaptation (RA) is a fundamental mechanism in 802.11 systems. It allows transmitters to adapt the coding and modulation scheme as well as the MIMO transmission mode to the radio channel conditions, and in turn, to learn and track the (mode, rate) pair providing the highest throughput. So far, the design of RA mechanisms has been mainly driven by heuristics. In contrast, in this paper, we rigorously formulate such design as an online stochastic optimisation problem. We solve this problem and present ORS (Optimal Rate Sampling), a family of (mode, rate) pair adaptation algorithms that provably learn as fast as it is possible the best pair for transmission. We study the performance of ORS algorithms in both stationary radio environments where the successful packet transmission probabilities at the various (mode, rate) pairs do not vary over time, and in non-stationary environments where these probabilities evolve. We show that under ORS algorithms, the throughput loss due to the need to explore sub-optimal (mode, rate) pairs does not depend on the number of available pairs, which is a crucial advantage as evolving 802.11 standards offer an increasingly large number of (mode, rate) pairs. We illustrate the efficiency of ORS algorithms (compared to the state-of-the-art algorithms) using simulations and traces extracted from 802.11 test-beds.

show abstract

Section: Stochastic Mab Problemsmentioning

confidence: 99%

Optimal Rate Sampling in 802.11 systems

Combes

Proutière

Yun

et al. 2014

IEEE INFOCOM 2014 - IEEE Conference on Computer Communications

View full text Add to dashboard Cite

show abstract

“…Most works in non-stationary k-armed bandit problems try to detect when the change in the distributions occurs, and then to re-learn with classical stationary approaches [45].…”

Section: The Prq-learning Algorithmmentioning

confidence: 99%

Learning domain structure through probabilistic policy reuse in reinforcement learning

Fernández

Veloso

2012

Prog Artif Intell

View full text Add to dashboard Cite

Policy Reuse is a transfer learning approach to improve a reinforcement learner with guidance from previously learned similar action policies. The method uses the past policies as a probabilistic bias where the learner chooses among the exploitation of the ongoing learned policy, the exploration of random unexplored actions, and the exploitation of past policies. In this work we demonstrate that Policy Reuse further contributes to the learning of the structure of a domain. Interestingly and almost as a side effect, Policy Reuse identifies classes of similar policies revealing a basis of core-policies of the domain. We demonstrate theoretically that, under a set of conditions to be satisfied, reusing such a set of core-policies allows us to bound the minimal expected gain received while learning a new policy. In general, Policy Reuse contributes to the overall goal of lifelong reinforcement learning, as (i) it incrementally builds a policy library; (ii) it provides a mechanism to reuse past policies; and (iii) it learns an abstract domain structure in terms of corepolicies of the domain.

show abstract

“…For example, the discounted/sliding-window UCB algorithm [25] assumes the nature of the non-stationarity is that the reward distribution is piece-wise and the number of changes is known. Similarly [27] makes the easier piecewise assumption, and also that the retrospective rewards for un-pulled arms are available -but they are not in active learning. In [28], the authors proposed to measure the total statistical variance of the consecutive distributions at each time interval.…”

Section: Existing Mab Ensembles Are Not Robust To Non-stationaritymentioning

confidence: 99%

Dynamic Ensemble Active Learning: A Non-Stationary Bandit with Expert Advice

Pang

Dong

et al. 2018

2018 24th International Conference on Pattern Recognition (ICPR)

View full text Add to dashboard Cite

Active learning aims to reduce annotation cost by predicting which samples are useful for a human teacher to label. However it has become clear there is no best active learning algorithm. Inspired by various philosophies about what constitutes a good criteria, different algorithms perform well on different datasets. This has motivated research into ensembles of active learners that learn what constitutes a good criteria in a given scenario, typically via multi-armed bandit algorithms. Though algorithm ensembles can lead to better results, they overlook the fact that not only does algorithm efficacy vary across datasets, but also during a single active learning session. That is, the best criteria is non-stationary. This breaks existing algorithms' guarantees and hampers their performance in practice. In this paper, we propose dynamic ensemble active learning as a more general and promising research direction. We develop a dynamic ensemble active learner based on a non-stationary multi-armed bandit with expert advice algorithm. Our dynamic ensemble selects the right criteria at each step of active learning. It has theoretical guarantees, and shows encouraging results on 13 popular datasets.

show abstract

Piecewise-stationary bandit problems with side observations

Cited by 65 publications

References 15 publications

Optimal Rate Sampling in 802.11 systems

Optimal Rate Sampling in 802.11 systems

Learning domain structure through probabilistic policy reuse in reinforcement learning

Dynamic Ensemble Active Learning: A Non-Stationary Bandit with Expert Advice

Contact Info

Product

Resources

About