Two-Stage Constrained Actor-Critic for Short Video Recommendation

Cai, Qingpeng; Xue, Zhenghai; Zhang, Chi; Xue, Wanqi; Liu, Shuchang; Zhan, Ruohan; Wang, Xueliang; Zuo, Tianyou; Xie, Wenhui; Zheng, Dong; Jiang, Peng; Gai, Kun

doi:10.48550/arxiv.2302.01680

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A desired recommendation policy should be able to satisfy users for a long time [45]. Therefore, it is natural to involve Reinforcement Learning (RL) which is a type of Machine Learning concerned with how an intelligent agent can take actions to pursue a longterm goal [1,4,5,35]. In this setting, the recommendation process is formulated as a sequential decision process where the recommender interacts with users and receives users' online feedback (i.e., rewards) to optimize users' long-term engagement, rather than fitting a model on a set of samples based on supervised learning [7,53,62].…”

Section: Introductionmentioning

confidence: 99%

Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation

Gao

Huang

Chen

et al. 2023

Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Offline reinforcement learning (RL), a technology that offline learns a policy from logged data without the need to interact with online environments, has become a favorable choice in decision-making processes like interactive recommendation. Offline RL faces the value overestimation problem. To address it, existing methods employ conservatism, e.g., by constraining the learned policy to be close to behavior policies or punishing the rarely visited state-action pairs. However, when applying such offline RL to recommendation, it will cause a severe Matthew effect, i.e., the rich get richer and the poor get poorer, by promoting popular items or categories while suppressing the less popular ones. It is a notorious issue that needs to be addressed in practical recommender systems.In this paper, we aim to alleviate the Matthew effect in offline RL-based recommendation. Through theoretical analyses, we find that the conservatism of existing methods fails in pursuing users' long-term satisfaction. It inspires us to add a penalty term to relax the pessimism on states with high entropy of the logging policy and indirectly penalizes actions leading to less diverse states. This leads to the main technical contribution of the work: Debiased model-based Offline RL (DORL) method. Experiments show that DORL not only captures user interests well but also alleviates the Matthew effect. The implementation is available via https://github. com/chongminggao/DORL-codes. CCS CONCEPTS• Information systems → Recommender systems.

show abstract