Sliding-Window Thompson Sampling for Non-Stationary Settings

Trovò, Francesco; Paladino, Stefano; Restelli, Marcello; Gatti, Nicola

doi:10.1613/jair.1.11407

Cited by 35 publications

(28 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The MAB algorithm we propose, named f-Discounted-Sliding-Window Thompson Sampling (f-dsw TS) , mixes two different approaches: a discount approach, inspired by the work of [ 36 , 37 ] and a sliding window approach, as in [ 15 , 37 ]. Our solution implicitly assigns more relevance to the recent evidences from the environment, as in the sliding window approach, given that these rewards are not discounted.…”

Section: Methodsmentioning

confidence: 99%

“… Discounted TS : the TS enhanced with a discount factor, presented in [ 36 ]: the parameter

controls the amount of discount. Sliding Window TS : the TS with a global sliding window, presented in [ 15 ]. The parameter n controls the size of the sliding window.…”

Section: Methodsmentioning

confidence: 99%

“…The sliding window ensures that the agent can adapt faster to the potential concept drift, because it takes into account only the last n rewards for estimating the payoff of each arm. Unlike [ 15 ], that exploits a single window for all the actions available to the agent, regardless of the arm that was played, we made the decision to use one window for each arm. Thus, we keep K different lists with the last n actions played, on which to update the estimations of the agent over the different reward distributions.…”

Section: Introductionmentioning

confidence: 99%

“…The two approaches inspiring the f-dsw TS algorithm are presented by Raj et al [ 36 ] and Trovo et al [ 15 ]. In the first one, the authors propose a Discounted Thompson Sampling (dTS) [ 36 ], which adds a discount to systematically increase the variance of the prior distribution and maintain exploration during time, reducing the effect of past observations.…”

Section: Introductionmentioning

confidence: 99%

“…In the first one, the authors propose a Discounted Thompson Sampling (dTS) [ 36 ], which adds a discount to systematically increase the variance of the prior distribution and maintain exploration during time, reducing the effect of past observations. Instead, Sliding Window Thompson Sampling (SW-TS) [ 15 ] uses a sliding window approach to tackle, in a unified fashion, two different forms of non-stationarity: abruptly changing and incremental changing. This method exploits a sliding window of size n in order to process only the most recent n actions performed by the agent for the reward estimation.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Non Stationary Multi-Armed Bandit: Empirical Evaluation of a New Concept Drift-Aware Algorithm

Cavenaghi

Sottocornola

Stella

et al. 2021

Entropy

View full text Add to dashboard Cite

The Multi-Armed Bandit (MAB) problem has been extensively studied in order to address real-world challenges related to sequential decision making. In this setting, an agent selects the best action to be performed at time-step t, based on the past rewards received by the environment. This formulation implicitly assumes that the expected payoff for each action is kept stationary by the environment through time. Nevertheless, in many real-world applications this assumption does not hold and the agent has to face a non-stationary environment, that is, with a changing reward distribution. Thus, we present a new MAB algorithm, named f-Discounted-Sliding-Window Thompson Sampling (f-dsw TS), for non-stationary environments, that is, when the data streaming is affected by concept drift. The f-dsw TS algorithm is based on Thompson Sampling (TS) and exploits a discount factor on the reward history and an arm-related sliding window to contrast concept drift in non-stationary environments. We investigate how to combine these two sources of information, namely the discount factor and the sliding window, by means of an aggregation function f(.). In particular, we proposed a pessimistic (f=min), an optimistic (f=max), as well as an averaged (f=mean) version of the f-dsw TS algorithm. A rich set of numerical experiments is performed to evaluate the f-dsw TS algorithm compared to both stationary and non-stationary state-of-the-art TS baselines. We exploited synthetic environments (both randomly-generated and controlled) to test the MAB algorithms under different types of drift, that is, sudden/abrupt, incremental, gradual and increasing/decreasing drift. Furthermore, we adapt four real-world active learning tasks to our framework—a prediction task on crimes in the city of Baltimore, a classification task on insects species, a recommendation task on local web-news, and a time-series analysis on microbial organisms in the tropical air ecosystem. The f-dsw TS approach emerges as the best performing MAB algorithm. At least one of the versions of f-dsw TS performs better than the baselines in synthetic environments, proving the robustness of f-dsw TS under different concept drift types. Moreover, the pessimistic version (f=min) results as the most effective in all real-world tasks.

show abstract

Section: Methodsmentioning

confidence: 99%

“… Discounted TS : the TS enhanced with a discount factor, presented in [ 36 ]: the parameter

controls the amount of discount. Sliding Window TS : the TS with a global sliding window, presented in [ 15 ]. The parameter n controls the size of the sliding window.…”

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Non Stationary Multi-Armed Bandit: Empirical Evaluation of a New Concept Drift-Aware Algorithm

Cavenaghi

Sottocornola

Stella

et al. 2021

Entropy

View full text Add to dashboard Cite

show abstract

Research on hyper‐level of hyper‐heuristic framework for MOTCP

Guo,

Wang,

Han

et al. 2023

Software Testing Verif & Rel

View full text Add to dashboard Cite

SummaryHeuristic algorithms are widely used to solve multi‐objective test case prioritization (MOTCP) problems. However, they perform differently for different test scenarios, which conducts difficulty in applying a suitable algorithm for new test requests in the industry. A concrete hyper‐heuristic framework for MOTCP (HH‐MOTCP) is proposed for addressing this problem. It mainly has two parts: low‐level encapsulating various algorithms and hyper‐level including an evaluation and selection mechanism that dynamically selects low‐level algorithms. This framework performs good but still difficult to keep in the best three. If the evaluation mechanism can more accurately analyse the current results, it will help the selection strategy to find more conducive algorithms for evolution. Meanwhile, if the selection strategy can find a more suitable algorithm for the next generation, the performance of the HH‐MOTCP framework will be better. In this paper, we first propose new strategies for evaluating the current generation results, then perform an extensive study on the selection strategies which decide the heuristic algorithm for the next generation. Experimental results show that the new evaluation and selection strategies proposed in this paper can make the HH‐MOTCP framework more effective and efficient, which makes it almost the best two except for one test object and ahead in about 40% of all test objects.

show abstract

Exploiting History Data for Nonstationary Multi-armed Bandit

Chiusano

Trovò

et al. 2021

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

The Multi-armed Bandit (MAB) framework has been applied successfully in many application fields. In the last years, the use of active approaches to tackle the nonstationary MAB setting, i.e., algorithms capable of detecting changes in the environment and re-configuring automatically to the change, has been widening the areas of application of MAB techniques. However, such approaches have the drawback of not reusing information in those settings where the same environment conditions recur over time. This paper presents a framework to integrate past information in the abruptly changing nonstationary setting, which allows the active MAB approaches to recover from changes quickly. The proposed framework is based on well-known break-point prediction methods to correctly identify the instant the environment changed in the past, and on the definition of recurring concepts specifically for the MAB setting to reuse information from recurring MAB states, when necessary. We show that this framework does not change the order of the regret suffered by the active approaches commonly used in the bandit field. Finally, we provide an extensive experimental analysis on both synthetic and real-world data, showing the improvement provided by our framework.

show abstract

Sliding-Window Thompson Sampling for Non-Stationary Settings

Cited by 35 publications

References 18 publications

Non Stationary Multi-Armed Bandit: Empirical Evaluation of a New Concept Drift-Aware Algorithm

Non Stationary Multi-Armed Bandit: Empirical Evaluation of a New Concept Drift-Aware Algorithm

Research on hyper‐level of hyper‐heuristic framework for MOTCP

Exploiting History Data for Nonstationary Multi-armed Bandit

Contact Info

Product

Resources

About