Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems

Abstract

Recommender systems play a crucial role in our daily lives. Feed streaming mechanism has been widely used in the recommender system, especially on the mobile Apps. The feed streaming setting provides users the interactive manner of recommendation in never-ending feeds. In such a manner, a good recommender system should pay more attention to user stickiness, which is far beyond classical instant metrics and typically measured by long-term user engagement. Directly optimizing long-term user engagement is a non-trivial problem, as the learning target is usually not available for conventional supervised learning methods. Though reinforcement learning (RL) naturally fits the problem of maximizing the long term rewards, applying RL to optimize long-term user engagement is still facing challenges: user behaviors are versatile to model, which typically consists of both instant feedback (e.g., clicks) and delayed feedback (e.g., dwell time, revisit); in addition, performing effective off-policy learning is still immature, especially when combining bootstrapping and function approximation.To address these issues, in this work, we introduce a RL framework -FeedRec to optimize the long-term user engagement. Fee-dRec includes two components: 1) a Q-Network which designed in hierarchical LSTM takes charge of modeling complex user behaviors, and 2) a S-Network, which simulates the environment, assists the Q-Network and voids the instability of convergence in policy learning. Extensive experiments on synthetic data and a real-world large scale data show that FeedRec effectively optimizes the longterm user engagement and outperforms state-of-the-arts. CCS CONCEPTS• Information systems → Recommender systems; Personalization; • Theory of computation → Sequential decision making. * Work performed during an internship at JD.com.

Neural Interactive Collaborative Filtering

“…In this work, the exploration policy is parameterized with multichannel stacked self-attention neural networks, which separately capture the information of versatile user behaviors since different rewarding recommendations for a specific user are usually extremely imbalanced (e.g., liking items usually are much fewer than disliking items) [18,45,48]. In Figure 2, we presented the neural architecture for exploration policy, which consists of an embedding layer, self-attentive blocks, and a policy layer.…”

Learning to Collaborate in Multi-Module Recommendation via Multi-Agent Reinforcement Learning without Communication

“…DRN updates periodically after obtaining long term-reward such as return time [29]. An algorithm is proposed to use two individual LSTM modules for items with short-term and long-term rewards respectively [30]. The diversity of recommended sets is added to the reward function [15].…”

Deep Multifaceted Transformers for Multi-objective Ranking in Large-Scale E-commerce Recommender Systems

“…Recommender Systems, which aim to recommend potentially interested items for users and solve the information explosion problem, are playing critical roles in E-commerce sites (e.g., Amazon, JD.com, Alibaba) [10,20,41,42], videos sharing sites (e.g., YouTube) [7], picture sharing sites (e.g., Pinterest) [36], social networks (e.g., Facebook) [11,13] and so on. For example, in JD.com, one of the largest E-commerce sites in the world, the Recommender System serves more than 0.3 billion users in China, Thailand, Malaysia and other countries, and contributes billions of dollars for the Gross Merchandise Volume (GMV) (i.e., the total sales value for merchandise sold) each year.…”

“…Many ranking methods for recommendations have been proposed, including tree-based methods [9], deep neural networks [5,7,10,40,41], reinforcement learning [42,43] based models and so on. However, designing a real-world large-scale E-commerce recommender system still faces many challenges, including:…”