Learning to Collaborate in Multi-Module Recommendation via Multi-Agent Reinforcement Learning without Communication

Abstract

With the rise of online e-commerce platforms, more and more customers prefer to shop online. To sell more products, online platforms introduce various modules to recommend items with different properties such as huge discounts. A web page often consists of different independent modules. The ranking policies of these modules are decided by different teams and optimized individually without cooperation, which might result in competition between modules. Thus, the global policy of the whole page could be sub-optimal. In this paper, we propose a novel multi-agent cooperative reinforcement learning approach with the restriction that different modules cannot communicate. Our contributions are threefold. Firstly, inspired by a solution concept in game theory named correlated equilibrium, we design a signal network to promote cooperation of all modules by generating signals (vectors) for different modules. Secondly, an entropy-regularized version of the signal network is proposed to coordinate agentsâȂŹ exploration of the optimal global policy. Furthermore, experiments based on real-world e-commerce data demonstrate that our algorithm obtains superior performance over baselines. CCS CONCEPTS • Information systems → Recommender systems; • Computing methodologies → Multi-agent systems.

Learning from Atypical Behavior: Temporary Interest Aware Recommendation Based on Reinforcement Learning

“…Generate the prototype item a p t and the recommended item a r t using Equations ( 6) and (7), respectively; Compute the losses L q , L r,c based on the minibatch, using Equations ( 17) and (20), respectively;…”

“…RL-based models aim to learn an optimal strategy to maximize the long terms rewards. RL-based models can be divided into three categories, the policy-based methods [4], [33], [31], [1], [12], [2], [30], the value-based methods [32], [34], [37], [11], and the actor-critic based methods [28], [6], [5], [7], [29], [24]. Chen et al [1] propose to use a balanced hierarchical clustering tree to tackle the large action space problem.…”

“…Zhou et al [35] propose to obtain item representations through a graph convolution network and obtain users' state embedding vectors through a recurrent neural network, and then trains the recommendation strategy with the Q-leaning algorithm. He et al [7] and Feng et al [5] treat each agent as a scenario and use a multi-agent framework to improve the total performance of all scenarios. Zhang et al [28] apply a multi-agent reinforcement learning method to coauthor network analysis for the dynamic collaboration recommendation.…”

“…To address the above challenges, motivated by the success of the reinforcement learning (RL) based recommendation models [32], [15], [36], [17], [5], [9], [7], we propose a novel RL-based model, called Temporary Interest Aware Recommendation (TIARec), which is able to distinguish atypical interactions from normal ones without annotations and capture the temporary interest as well as the general preference. TIARec contains a recommender agent and an auxiliary classifier agent.…”

Reinforced KGs reasoning for explainable sequential recommendation

“…In recent years, to promoting the recommendation models to search meaningful paths rather than enumerate all possible paths in KGs, RL has been gradually introduced in recommendations. Some RL-based recommendation models [4,9,41] have achieved outstanding performance in recommendation. For example, Song et al [20] proposed a knowledge-aware recommendation model to generates meaningful paths from users to relevant items by learning a walking policy on the user-item-entity graph, which is designed to deal with the data sparsity and cold start problems.…”