Pseudo Dyna-Q

Zou

Proceedings of the ACM Web Conference 2023

et al. 2023

Self Cite

Improving user retention with reinforcement learning (RL) has attracted increasing attention due to its significant importance in boosting user engagement. However, training the RL policy from scratch without hurting users' experience is unavoidable due to the requirement of trial-and-error searches. Furthermore, the offline methods, which aim to optimize the policy without online interactions, suffer from the notorious stability problem in value estimation or unbounded variance in counterfactual policy evaluation. To this end, we propose optimizing user retention with Decision Transformer (DT), which avoids the offline difficulty by translating the RL as an autoregressive problem. However, deploying the DT in recommendation is a non-trivial problem because of the following challenges: (1) deficiency in modeling the numerical reward value; (2) data discrepancy between the policy learning and recommendation generation; (3) unreliable offline performance evaluation. In this work, we, therefore, contribute a series of strategies for tackling the exposed issues. We first articulate an efficient reward prompt by weighted aggregation of meta embeddings for informative reward embedding. Then, we endow a weighted contrastive learning method to solve the discrepancy between training and inference. Furthermore, we design two robust offline metrics to measure user retention. Finally, the significant improvement in the benchmark datasets demonstrates the superiority of the proposed method. The implementation code is available at https://github.com/kesenzhao/DT4Rec.git. CCS CONCEPTS• Information Systems → Recommender Systems.

Section: Reinforcement Learning Based Recommendationmentioning

confidence: 99%

User Retention-oriented Recommendation with Decision Transformer

Zou

Proceedings of the ACM Web Conference 2023

et al. 2023

Self Cite

Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

“…Model-based RL. These methods directly model the environment dynamics [2,8,45], which improves sample efficiency. However, it is very difficult to estimate the state transition in many real-world recommendation tasks with large state and action spaces [1].…”

Section: Related Workmentioning

confidence: 99%

“…However, it is very challenging to apply RL to industrial-scale recommender systems serving a tremendous amount of users with diverse preferences concerning a huge item corpus [5]. As RL-based recommender systems often treat users as states and items as actions, the state and action spaces are extremely large (typically in millions or billions), making classic RL methods rather sample inefficient [2,8,45]. In classic RL scenarios, one way to improve sample efficiency is to adopt model-based RL, which directly models the environment dynamics [20,30].…”

Section: Introductionmentioning

confidence: 99%

Model-free Reinforcement Learning with Stochastic Reward Stabilization for Recommender Systems

Cai¹,

Shenliao²,

Jiang³

et al. 2023

Model-free RL-based recommender systems have recently received increasing research attention due to their capability to handle partial feedback and long-term rewards. However, most existing research has ignored a critical feature in recommender systems: one user's feedback on the same item at different times is random. The stochastic rewards property essentially differs from that in classic RL scenarios with deterministic rewards, which makes RL-based recommender systems much more challenging. In this paper, we first demonstrate in a simulator environment where using direct stochastic feedback results in a significant drop in performance. Then to handle the stochastic feedback more efficiently, we design two stochastic reward stabilization frameworks that replace the direct stochastic feedback with that learned by a supervised model. Both frameworks are model-agnostic, i.e., they can effectively utilize various supervised models. We demonstrate the superiority of the proposed frameworks over different RL-based recommendation baselines with extensive experiments on a recommendation simulator as well as an industrial-level recommender system. CCS CONCEPTS• Computing methodologies → Sequential decision making.

“…Relating buyers' impression to a potential seller via buyer used search words [28]. Similar use cases of relating customer feedback to items through Q-learning and self-attention mechanism [29]. Graph scenarios have been approached using a knowledge graph to discover high-quality negative signals from implicit feedback to interpret user intent [30].…”

Section: Related Workmentioning

confidence: 99%

Reinforcement Learning Page Prediction for Hierarchically Ordered Municipal Websites

et al. 2021

Public websites offer information on a variety of topics and services and are accessed by users with varying skills to browse the kind of electronic document repositories. However, the complex website structure and diversity of web browsing behavior create a challenging task for click prediction. This paper presents the results of a novel reinforcement learning approach to model user browsing patterns in a hierarchically ordered municipal website. We study how accurate predictor the browsing history is, when the target pages are not immediate next pages pointed by hyperlinks, but appear a number of levels down the hierarchy. We compare traditional type of baseline classifiers’ performance against our reinforcement learning-based training algorithm.