Whole-Chain Recommendations

Abstract

With the recent prevalence of Reinforcement Learning (RL), there have been tremendous interests in developing RL-based recommender systems. In practical recommendation sessions, users will sequentially access multiple scenarios, such as the entrance pages and the item detail pages, and each scenario has its specific characteristics. However, the majority of existing RL-based recommender systems focus on optimizing one strategy for all scenarios or separately optimizing each strategy, which could lead to sub-optimal overall performance. In this paper, we study the recommendation problem with multiple (consecutive) scenarios, i.e., whole-chain recommendations. We propose a multi-agent RL-based approach (DeepChain), which can capture the sequential correlation among different scenarios and jointly optimize multiple recommendation strategies. To be specific, all recommender agents (RAs) share the same memory of users' historical behaviors, and they work collaboratively to maximize the overall reward of a session. Note that optimizing multiple recommendation strategies jointly faces two challenges in the existing model-free RL model [22]-(i) it requires huge amounts of user behavior data, and (ii) the distribution of reward (users' feedback) are extremely unbalanced. In this paper, we introduce model-based RL techniques to reduce the training data requirement and execute more accurate strategy updates. The experimental results based on a real e-commerce platform demonstrate the effectiveness of the proposed framework.

A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions

“…Work Value-based [18,96,119,135] Policy-based [4,38] Hybrid [125] Policy-based methods. IRecGAN [4] is a model-based method that adopts generative adversarial training to improve the robustness of policy learning.…”

“…Hybrid method can be recognized as a midpoint between value-based and policy gradient-based methods. DeepChain [125] uses the multi-agent setting to relieve the sub-optimality problem. The sub-optimality problem is caused by the one for all setting that optimizes one policy for all users.…”

“…As mentioned, the transition probability function is normally equivalent to users' behavior probability which is hard to determine in a recommender system. Hence, existing works [4,18,96,119,125,135] approximate P using a neural network or embedding it into the world model. Zhao et al [125] design a probability network to estimate P while [4,18] uses a GAN to generate user behavior where P is embedded in the latent space.…”

“…Hence, existing works [4,18,96,119,125,135] approximate P using a neural network or embedding it into the world model. Zhao et al [125] design a probability network to estimate P while [4,18] uses a GAN to generate user behavior where P is embedded in the latent space. Different from them, [96,135] relies on the world model to predict users' next behavior and feed it into the policy learning process.…”

AutoLoss: Automated Loss Function Search in Recommendations

“…In the era of information explosion, recommender systems play a pivotal role in alleviating information overload, which vastly enhance user experiences in many commercial applications, such as generating playlists in video and music services [56,64], recommending products in online stores [8,60,61,63,66], and suggesting locations for geo-social events [15,35,62]. With the recent growth of deep learning techniques, there have been increasing interests in developing deep recommender systems (DRS) [37,50].…”