Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning

Abstract

With the recent prevalence of Reinforcement Learning (RL), there have been tremendous interests in developing RL-based recommender systems. In practical recommendation sessions, users will sequentially access multiple scenarios, such as the entrance pages and the item detail pages, and each scenario has its own recommendation strategy. However, the majority of existing RL-based recommender systems focus on optimizing one strategy for all scenarios or separately optimizing each strategy, which could lead to sub-optimal overall performance. In this paper, we study the recommendation problem with multiple (consecutive) scenarios, i.e., whole-chain recommendations. We propose a multi-agent reinforcement learning based approach (DeepChain), which can capture the sequential correlation among different scenarios and jointly optimize multiple recommendation strategies. To be specific, all recommender agents share the same memory of users' historical behaviors, and they work collaboratively to maximize the overall reward of a session. Note that optimizing multiple recommendation strategies jointly faces two challenges in existing model-free RL model [10]-(i) it requires huge amounts of user behavior data, and (ii) the distribution of reward (users' feedback) are extremely unbalanced. In this paper, we introduce model-based reinforcement learning techniques to reduce the training data requirement and execute more accurate strategy updates. The experimental results based on a real e-commerce platform demonstrate the effectiveness of the proposed framework.

Graph Neural Networks for Social Recommendation

“…In recent years, deep neural network models had a great impact on learning effective feature representations in various fields, such as speech recognition [12], Computer Vision (CV) [14] and Natural Language Processing (NLP) [4]. Some recent efforts have applied deep neural networks to recommendation tasks and shown promising results [41], but most of them used deep neural networks to model audio features of music [32], textual description of items [3,33], and visual content of images [40]. Besides, NeuMF [11] presented a Neural Collaborative Filtering framework to learn the non-linear interactions between users and items.…”

Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems

“…Equation (5); 30 end 31 # Updating the S-Network. 32 for j = 1 : K do 33 Sample mini-batches of (s t , i t , r t , s t +1 ) from M; 34 Set f , d , l , r according to r t , s t +1 ; 35 Update θ s via mini-batch SGD w.r.t. the loss in Equation (7); will cause the selection base.…”

“…Unfortunately, current methods including Monte Carlo (MC) and temporaldifference (TD) have limitations for offline policy learning in realistic recommender systems: MC-based methods suffer from the problem of high variance, especially when facing enormous action space (e.g., billions of candidate items) in real-world applications; TD-based methods improve the efficiency by using bootstrapping techniques in estimation, which, however, is confronted with another notorious problem called Deadly Triad (i.e., the problem of instability and divergence arises whenever combining function approximation, bootstrapping, and offline training [24]). Unfortunately, state-of-the-art methods [33,34] in recommender systems, which are designed with neural architectures, will encounter inevitably the Deadly Triad problem in offline policy learning.…”

The Disruptions of 5G on Data-Driven Technologies and Applications

“…This is possible because (i) edge storage and compute resources are more powerful with various system-on-chip (SoC) technologies and (ii) there is a dataprivacy practice to keep personal data locally. Further, due to its inherent capability of adaptive modeling and longterm planning, reinforcement learning presents potential in building interactive and personalized models, such as interactive recommendation systems [111], [112], [113].…”