State representation modeling for deep reinforcement learning based recommendation

Liu, Feng; Tang, Ruiming; Li, Xutao; Zhang, Weinan; Ye, Yunming; Chen, Haokun; Guo, Huifeng; Zhang, Yuzhou; He, Xiuqiang

doi:10.1016/j.knosys.2020.106170

Cited by 44 publications

(58 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Existing studies usually directly use the embedding as the state representation. Liu et al [58,61] propose a supervised learning method to generate a better state representation by utilizing an attention mechanism and a pooling operation as shown in Figure 7. Such a representation method requires training a representation network when training the main policy network, which increases the model complexity.…”

Section: State Representationmentioning

confidence: 99%

See 1 more Smart Citation

A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions

Chen¹,

Yao²,

McAuley³

et al. 2021

Preprint

View full text Add to dashboard Cite

In light of the emergence of deep reinforcement learning (DRL) in recommender systems research and several fruitful results in recent years, this survey aims to provide a timely and comprehensive overview of the recent trends of deep reinforcement learning in recommender systems. We start with the motivation of applying DRL in recommender systems. Then, we provide a taxonomy of current DRL-based recommender systems and a summary of existing methods. We discuss emerging topics and open issues, and provide our perspective on advancing the domain. This survey serves as introductory material for readers from academia and industry into the topic and identifies notable opportunities for further research.

show abstract

Section: State Representationmentioning

confidence: 99%

“…7. State representation used in works [58,61]. ℎ 𝑡 is the output of an attention layer that takes the representation of users' history at time 𝑡 as input and 𝑔(•) is the pooling operation.…”

Section: Attention Layermentioning

confidence: 99%

A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions

Chen¹,

Yao²,

McAuley³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…An interesting approach is to define states as the cluster resulted from the coclustering or biclustering of users and items [6] or to extend the state to include user demographics [5]. Efforts are as well invested in how to best represent the state in a RL RecSys [61]. Currently, and as to the knowledge of the author, in the current literature there is no approach where the recommendation context, user demographics, behavioral patterns and recent browsing/interaction history is taken into account in the state definition.…”

Section: F Recommender Systems Using Reinforcement Learningmentioning

confidence: 99%

“…Actions are mostly defined as selecting an item to be recommended from the whole discrete action space which contains the candidate items [12,14,15] or even whether to give a recommendation or not, and if yes, what would be the item to recommend [13]. There are authors that consider recommending a list of items [5,11,61]. One of the most different approaches it to recommend items from neighboring clusters to the user-items one [6].…”

Section: F Recommender Systems Using Reinforcement Learningmentioning

confidence: 99%

Fairness Embedded Adaptive Recommender System: A Conceptual Framework

Popa¹

2021

IJACSA

View full text Add to dashboard Cite

In the current fast paced and constantly changing environment, companies should ensure that their way of interacting with user is both relevant and highly adaptive. In order to stay competitive, companies should invest in state-ofthe-art technologies that optimize the relationship with the user using increasingly available data. The most popular applications used to develop user relationship are Recommender Systems. The vast majority of the traditional recommender system considers recommendation as a static procedure and focus on a specific type of recommendation, being not very agile in adapting to new situations. Also, when implementing a Recommender System there is the need to ensure fairness in the way decisions are made upon customer data. In this paper, it is proposed a novel Reinforcement Learning-based recommender system that is highly adaptive to changes in customer behavior and focuses on ensuring both producer and consumer fairness, Fairness Embedded Adaptive Recommender System (FEARS). The approach overcomes Reinforcement Learning's main drawback in recommendation area by using a small, but meaningful action space. Also, there are presented two fairness metrics, their calculation and adaptation for usage with Reinforcement Learning, this way ensuring that the system gets to the optimal trade-off between personalization and fairness.

show abstract

“…There are several difficulties in building RLRS, such as the state representation (Lei et al 2020;Liu et al 2020;Wang et al 2020b), the slated-based recommendation setting (Sunehag et al 2015;Swaminathan et al 2017;Chen et al 2019;Gong et al 2019;Ie et al 2019) and the largescale recommendation problem (Dulac-Arnold et al 2015;Zhao et al 2019). However, there has been limited attention paid to the negative impact of offline setting, i.e., training RLRS agents from static datasets.…”

Section: Related Workmentioning

confidence: 99%

Value Penalized Q-Learning for Recommender Systems

Gao,

Xu,

Zhao

2021

Preprint

View full text Add to dashboard Cite

Scaling reinforcement learning (RL) to recommender systems (RS) is promising since maximizing the expected cumulative rewards for RL agents meets the objective of RS, i.e., improving customer's long-term satisfaction. A key approach to this goal is offline RL, which aims to learn policies from logged data. However, the high-dimensional action space and the non-stationary dynamics in commercial RS intensify distributional shift issues, making it challenging to apply offline RL methods to RS. To alleviate the action distribution shift problem in extracting RL policy from static trajectories, we propose Value Penalized Q-learning (VPQ), an uncertaintybased offline RL algorithm. It penalizes the unstable Q-values in the regression target by uncertainty-aware weights, without the need to estimate the behavior policy, suitable for RS with a large number of items. We derive the penalty weights from the variances across an ensemble of Q-functions. To alleviate distributional shift issues at test time, we further introduce the critic framework to integrate the proposed method with classic RS models. Extensive experiments conducted on two realworld datasets show that the proposed method could serve as a gain plugin for existing RS models.

show abstract

State representation modeling for deep reinforcement learning based recommendation

Cited by 44 publications

References 20 publications

A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions

A Survey of Deep Reinforcement Learning in Recommender Systems: A Systematic Review and Future Directions

Fairness Embedded Adaptive Recommender System: A Conceptual Framework

Value Penalized Q-Learning for Recommender Systems

Contact Info

Product

Resources

About