A survey on the bandit problem with switching costs

Jun, Tackseung

doi:10.1007/s10645-004-2477-z

Cited by 70 publications

(45 citation statements)

References 90 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is optimal to switch to another cash-flow stream as soon as the benefit of delaying the switch, as measured by an appropriate "adjoint variable," drops to zero. Switching costs, which in our problem tend to delay actions, have also been studied for the stochastic multi-armed bandit problem; see Jun (2004) for a survey.…”

Section: Literaturementioning

confidence: 99%

Optimal switching between cash-flow streams

Weber

2017

Math Meth Oper Res

View full text Add to dashboard Cite

We consider the problem of finding the optimal time to switch between two measurable cash-flow streams. A complete characterization of the set of solutions is obtained in terms of adjoint variables which measure the available gain from deviations. An iterative procedure for the computation of the adjoint variables is provided. The results are generalized to multiple switching times, multiple cash-flow streams, switching costs, as well as switch-triggered cash-flow streams that arise in equipmentreplacement problems.

show abstract

Section: Literaturementioning

confidence: 99%

Optimal switching between cash-flow streams

Weber

2017

Math Meth Oper Res

View full text Add to dashboard Cite

show abstract

“…Additive switching cost are common in the literature and algorithms exist (e.g., Banks and Sundaram 1994;Dushochet and Hongler 2003;Jun 2004), but additive switching costs impose path dependence and make it infeasible to determine when to morph in real time. On the other hand, a multiplicative switching cost can be factored out in a Bellman equation.…”

Section: Assumption 1 Switching Costsmentioning

confidence: 99%

Website Morphing 2.0: Switching Costs, Partial Exposure, Random Exit, and When to Morph

2014

View full text Add to dashboard Cite

January 2014Website morphing infers latent customer segments from clickstreams then changes websites' look and feel to maximize revenue. The established algorithm infers latent segments from a pre-set number of clicks and then selects the best "morph" using Expected Gittins' Indices. Switching costs, potential website exit, and all clicks prior to morphing are ignored.We model switching costs, potential website exit, and the (potentially differential) impact of all clicks to determine when to morph for each customer. Morphing earlier means more customer clicks are based on the optimal morph; morphing later reveals more about the customer's latent segment. We couple this within-customer optimization to between-customer Expected Gittins' Index optimization to determine which website "look and feel" to give to each customer at each click. We evaluate the improved algorithm with synthetic data and with a proof-of-feasibility application to Japanese bank card-loans. The proposed algorithm generalizes the established algorithm, is feasible in real time, performs substantially better when tuning parameters are identified from calibration data, and is reasonably robust to misspecification. Keywords:Automated Bayesian inference of latent cognitive-style segments with dynamic programming optimization to match website designs to customers. Bayesian inference on a customer's clickstream infers probabilities that the customer belongs to the latent segments. Using these probabilities and data from past purchases, the dynamic program automatically selects the best look and feel for the website for each customer. The "morph" assignment is (near) optimal in the sense that it balances learning about the best assignment for a segment with the profit that can be obtained by exploiting current knowledge of morph-to-segment purchase probabilities. HULB use data from a "calibration study" to simulate what would had happened had the BT Group implemented morphing on its broadband-sales website. HULB estimate that morphing would have increased revenue by $80M.Subsequently, Urban, Liberali, MacDonald, Bordley, and Hauser (2014) [hereafter Urban, et al.] adapted morphing to banner advertising. The only modification in HULB's algorithm was to account for multiple customer visits to the same website. In a field test with over 100,000 customers viewing over 450,000 banners on a CNET website, they report that banner morphing almost doubled click-through rates relative to a random assignment of banners. They also conducted a laboratory test on an automotive information-and-recommendation website to test the basic concept of morph-to-segment matching. The experiment replaced the automated algorithm with direct measurement in a 4-5 week longitudinal study.Click-through rates on banners, as well as consideration and preference for Chevrolet-branded vehicles, increased significantly when morphs were matched to customer segments. The Chevrolet application expanded the definitions of customer segments to include the stage of the automoti...

show abstract

“…MAB with switching costs was first considered in [17]. An excellent survey on MAB with switching costs can be found in [18].…”

Section: Related Workmentioning

confidence: 99%

Sequential Learning for Multi-Channel Wireless Network Monitoring With Channel Switching Costs

Szepesvári

Zheng

2014

IEEE Trans. Signal Process.

View full text Add to dashboard Cite

Abstract-We consider the problem of optimally assigning p sniffers to K channels to monitor the transmission activities in a multi-channel wireless network with switching costs. The activity of users is initially unknown to the sniffers and is to be learned along with channel assignment decisions to maximize the benefits of this assignment, resulting in the fundamental trade-off between exploration and exploitation. Switching costs are incurred when sniffers change their channel assignments. As a result, frequent changes are undesirable. We formulate the sniffer-channel assignment with switching costs as a linear partial monitoring problem, a super-class of multi-armed bandits. As the number of arms (sniffer-channel assignments) is exponential, novel techniques are called for, to allow efficient learning. We use the linear bandit model to capture the dependency amongst the arms and develop a policy that takes advantage of this dependency. We prove that the proposed Upper Confident Boundbased (UCB) policy enjoys a logarithmic regret bound in time t that depends sub-linearly on the number of arms, while its total switching cost grows in the order of O(log log(t)).

show abstract

A survey on the bandit problem with switching costs

Cited by 70 publications

References 90 publications

Optimal switching between cash-flow streams

Optimal switching between cash-flow streams

Website Morphing 2.0: Switching Costs, Partial Exposure, Random Exit, and When to Morph

Sequential Learning for Multi-Channel Wireless Network Monitoring With Channel Switching Costs

Contact Info

Product

Resources

About