Safe Policy Improvement with Baseline Bootstrapping

Laroche, Romain; Trichelair, Paul; Combes, Remi Tachet des

doi:10.48550/arxiv.1712.06924

Cited by 7 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other hand, for model-free methods, Sutton and Barto [23] identify a deadly triad of function approximation, bootstrapping, and off-policy learning. It emphasizes that function approximation equipped with Q-learning can diverge in the off-policy learning setting, making most off-policy RL methods very conservative in extrapolation because of the severe error-propagation issue [7,16]. In this work, we focus on the contextual bandit problem, which has many direct applications in recommender systems and online search.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Off-policy Bandits with Deficient Support

Sachdeva

Joachims

2020

Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

Learning effective contextual-bandit policies from past actions of a deployed system is highly desirable in many settings (e.g. voice assistants, recommendation, search), since it enables the reuse of large amounts of log data. State-of-the-art methods for such offpolicy learning, however, are based on inverse propensity score (IPS) weighting. A key theoretical requirement of IPS weighting is that the policy that logged the data has "full support", which typically translates into requiring non-zero probability for any action in any context. Unfortunately, many real-world systems produce support deficient data, especially when the action space is large, and we show how existing methods can fail catastrophically. To overcome this gap between theory and applications, we identify three approaches that provide various guarantees for IPS-based learning despite the inherent limitations of support-deficient data: restricting the action space, reward extrapolation, and restricting the policy space. We systematically analyze the statistical and computational properties of these three approaches, and we empirically evaluate their effectiveness. In addition to providing the first systematic analysis of support-deficiency in contextual-bandit learning, we conclude with recommendations that provide practical guidance. CCS CONCEPTS• Information systems → Retrieval models and ranking; • Computing methodologies → Learning from implicit feedback.

show abstract

Section: Related Workmentioning

confidence: 99%

“…For each fixed (u 1 , u 2 ) pair, the inner minimization objective is ERM with IPS where the reward is shifted by k = (u 1 − u 2 ). So, we can select k, solve (16), and compute the corresponding κ afterwards.…”

Section: Safe Learning By Restricting the Policy Spacementioning

confidence: 99%

Off-policy Bandits with Deficient Support

Sachdeva

Joachims

2020

Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

show abstract

“…These failure cases are hypothesized to be caused by erroneous generalization of the state-action value function (Q-value function) learned with function approximators, as suggested by Sutton (1995); Baird (1995); Tsitsiklis & Van Roy (1997); Van Hasselt et al (2018). To remedy this issue, two types of approaches have been proposed recently: 1) Agarwal et al (2019) proposes to apply a random ensemble of Q-value targets to stabilize the learned Q-function, 2) Fujimoto et al (2018a); ; Jaques et al (2019); Laroche & Trichelair (2017) propose to regularize the learned policy towards the behavior policy based on the intuition that unseen state-action pairs are more likely to receive overestimated Q-values. These proposed remedies have been shown to improve upon DQN or DDPG at performing policy improvement based on offline data.…”

Section: Introductionmentioning

confidence: 99%

Behavior Regularized Offline Reinforcement Learning

Wu,

Tucker,

Nachum

2019

Preprint

162

325

View full text Add to dashboard Cite

In reinforcement learning (RL) research, it is common to assume access to direct online interactions with the environment. However in many real-world applications, access to the environment is limited to a fixed offline dataset of logged experience. In such settings, standard RL algorithms have been shown to diverge or otherwise yield poor performance. Accordingly, recent work has suggested a number of remedies to these issues. In this work, we introduce a general framework, behavior regularized actor critic (BRAC), to empirically evaluate recently proposed methods as well as a number of simple baselines across a variety of offline continuous control tasks. Surprisingly, we find that many of the technical complexities introduced in recent methods are unnecessary to achieve strong performance. Additional ablations provide insights into which design choices matter most in the offline RL setting. 1 * Work performed while an intern at Google Brain.

show abstract

“…Model-free methods constitute the majority of offline RL algorithms in current RL literature. Among the first algorithms that considered the problem now known as offline RL -with no environment interaction and learning only from a static dataset that was collected under a baseline policy -was Safe Policy Improvement under Baseline Bootstrapping (SPIBB) (Laroche et al, 2017). It was designed for discrete actions and assumes that the baseline policy is passed to the learning algorithm as an input.…”

Section: Related Workmentioning

confidence: 99%

Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learning

Swazinna¹,

Udluft²,

Hein³

et al. 2022

Preprint

View full text Add to dashboard Cite

Offline reinforcement learning (RL) Algorithms are often designed with environments such as MuJoCo in mind, in which the planning horizon is extremely long and no noise exists. We compare model-free, model-based, as well as hybrid offline RL approaches on various industrial benchmark (IB) datasets to test the algorithms in settings closer to real world problems, including complex noise and partially observable states. We find that on the IB, hybrid approaches face severe difficulties and that simpler algorithms, such as rollout based algorithms or model-free algorithms with simpler regularizers perform best on the datasets.

show abstract

Safe Policy Improvement with Baseline Bootstrapping

Cited by 7 publications

References 0 publications

Off-policy Bandits with Deficient Support

Off-policy Bandits with Deficient Support

Behavior Regularized Offline Reinforcement Learning

Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learning

Contact Info

Product

Resources

About