Self-Supervised Reinforcement Learning for Recommender Systems

Abstract

In session-based or sequential recommendation, it is important to consider a number of factors like long-term user engagement, multiple types of user-item interactions such as clicks, purchases etc. The current state-of-the-art supervised approaches fail to model them appropriately. Casting sequential recommendation task as a reinforcement learning (RL) problem is a promising direction. A major component of RL approaches is to train the agent through interactions with the environment. However, it is often problematic to train a recommender in an on-line fashion due to the requirement to expose users to irrelevant recommendations. As a result, learning the policy from logged implicit feedback is of vital importance, which is challenging due to the pure off-policy setting and lack of negative rewards (feedback).In this paper, we propose self-supervised reinforcement learning for sequential recommendation tasks. Our approach augments standard recommendation models with two output layers: one for selfsupervised learning and the other for RL. The RL part acts as a regularizer to drive the supervised layer focusing on specific rewards (e.g., recommending items which may lead to purchases rather than clicks) while the self-supervised layer with cross-entropy loss provides strong gradient signals for parameter updates. Based on such an approach, we propose two frameworks namely Self-Supervised Qlearning (SQN) and Self-Supervised Actor-Critic (SAC). We integrate the proposed frameworks with four state-of-the-art recommendation models. Experimental results on two real-world datasets demonstrate the effectiveness of our approach. CCS CONCEPTS• Information systems → Recommender systems; Retrieval models and ranking; Novelty in information retrieval.

Self-Supervised Multi-Channel Hypergraph Convolutional Network for Social Recommendation

“…As the research of self-supervised learning is still in its infancy, there are only several works combining it with recommender systems [24,44,45,64]. These efforts either mine self-supervision signals from future/surrounding sequential data [24,45], or mask attributes of items/users to learn correlations of the raw data [64]. However, these thoughts cannot be easily adopted to social recommendation where temporal factors and attributes may not be available.…”

Fairness Embedded Adaptive Recommender System: A Conceptual Framework

“…States are defined in different ways in the existing literature. They can reflect a mapping of previous user-item interactions into a hidden state [15], user's recommendation and ad browsing history [13], previous items that a user clicked [12], the sequence of visited and recommended items [10] or a more detailed interaction sequence that contains clicking, purchasing, or skipping, leaving [14]. An interesting approach is to define states as the cluster resulted from the coclustering or biclustering of users and items [6] or to extend the state to include user demographics [5].…”

“…Actions are mostly defined as selecting an item to be recommended from the whole discrete action space which contains the candidate items [12,14,15] or even whether to give a recommendation or not, and if yes, what would be the item to recommend [13]. There are authors that consider recommending a list of items [5,11,61].…”

“…There is a series of publications that explore the usage of RL in the area of RecSys. Out of which there are those that focus on user-item interaction sequence or user's browsing history and use it to create a state that later is fed to the RL model [10][11][12][13][14][15]. A different approach is to use user and item sets which are obtained from bi-clustering as environmental states [6].…”

“…An earlier paper is using both user information and item information vectors and refers to it as context [16]. Important work on integrating negative influence of irrelevant recommendations is done by using negative rewards [12,13,15,17].…”