A Near-Optimal Change-Detection Based Algorithm for Piecewise-Stationary Combinatorial Semi-Bandits

Abstract

We investigate the piecewise-stationary combinatorial semi-bandit problem. Compared to the original combinatorial semi-bandit problem, our setting assumes the reward distributions of base arms may change in a piecewise-stationary manner at unknown time steps. We propose an algorithm, GLR-CUCB, which incorporates an efficient combinatorial semi-bandit algorithm, CUCB, with an almost parameter-free change-point detector, the Generalized Likelihood Ratio Test (GLRT). Our analysis shows that the regret of GLR-CUCB is upper bounded by O( √ N KT log T ), where N is the number of piecewise-stationary segments, K is the number of base arms, and T is the number of time steps. As a complement, we also derive a nearly matching regret lower bound on the order of Ω( √ N KT ), for both piecewise-stationary multi-armed bandits and combinatorial semi-bandits, using information-theoretic techniques and judiciously constructed piecewisestationary bandit instances. Our lower bound is tighter than the best available regret lower bound, which is Ω( √ T ). Numerical experiments on both synthetic and real-world datasets demonstrate the superiority of GLR-CUCB compared to other state-of-the-art algorithms.

Nearly Optimal Algorithms for Piecewise-Stationary Cascading Bandits

“…where the dependence on N is improved to In real-world applications, both L and T can be huge, for example, L and T are in the millions in web search, where the improvements are significant. Compared to recent works on piecewise-stationary MAB (Besson and Kaufmann, 2019) and combinatorial MAB (CMAB) (Zhou et al, 2019) that adopt GLRT as the change-point detector, the problem setting considered herein is different. In MAB, only one selected item rather than a list of items is allowed at each time.…”

“…Notice that although CMAB (Combes et al, 2015;Cesa-Bianchi and Lugosi, 2012;Chen et al, 2016) also allow a list of items each time, they have full feedback on all K items under semi-bandit setting. Furthermore, we develop the analysis of both UCB-based and KL-UCB based algorithms for CB, whereas only one of them (either UCB-based or KL-UCB based algorithm) is analyzed in Besson and Kaufmann (2019) and Zhou et al (2019). We also observe one interesting fact that the regret upper bounds of our proposed algorithms and minimax regret lower bounds match their counterparts in piecewise-stationary combinatorial semi-bandits (Zhou et al, 2019), in which the agent has access to the realizations of base arms in the played super arm.…”

“…Furthermore, we develop the analysis of both UCB-based and KL-UCB based algorithms for CB, whereas only one of them (either UCB-based or KL-UCB based algorithm) is analyzed in Besson and Kaufmann (2019) and Zhou et al (2019). We also observe one interesting fact that the regret upper bounds of our proposed algorithms and minimax regret lower bounds match their counterparts in piecewise-stationary combinatorial semi-bandits (Zhou et al, 2019), in which the agent has access to the realizations of base arms in the played super arm. Counterintuitively, this implies partial feedback need not increase the problem difficulty in this case.…”

“…(sketch). The main idea is to construct a randomized hard instance appropriate for the piecewise-stationary CB setting (see Zhou et al (2019) for a similar proof technique), in which at each time step there is only one item with highest click probability and the click probabilities of remaining items are the same. When the distribution change occurs, the best item changes uniformly at random.…”