Personalized Recommendation via Parameter-Free Contextual Bandits

Classical collaborative filtering, and content-based filtering methods try to learn a static recommendation model given training data. These approaches are far from ideal in highly dynamic recommendation domains such as news recommendation and computational advertisement, where the set of items and users is very fluid. In this work, we investigate an adaptive clustering technique for content recommendation based on exploration-exploitation strategies in contextual multi-armed bandit settings. Our algorithm takes into account the collaborative effects that arise due to the interaction of the users with the items, by dynamically grouping users based on the items under consideration and, at the same time, grouping items based on the similarity of the clusterings induced over the users. The resulting algorithm thus takes advantage of preference patterns in the data in a way akin to collaborative filtering methods. We provide an empirical analysis on medium-size real-world datasets, showing scalability and increased prediction performance (as measured by click-through rate) over state-of-the-art methods for clustering bandits. We also provide a regret analysis within a standard linear stochastic noise setting.

show abstract

“…Further work includes [25,32]. In [25], an ensemble of contextual bandits is used to address the cold-start problem in recommender systems.…”

Section: Related Workmentioning

confidence: 99%

Collaborative Filtering Bandits

Karatzoglou

Gentile

2016

Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval

250

159

View full text Add to dashboard Cite

show abstract

“…Another challenge is to sequentially recommend items to the target user in an online se ing and instantly adapt the up-to-date feedback to re ne the recommendation predictions. Multi-armed bandit algorithms [1,2,7,25] are widely applied to address the cold-start problem and balance the tradeo between exploration and exploitation in online recomemnder systems [16,24,29]. Interactive Collaborative Filtering (ICF) [32] tackles these problems in a partially online se ing leveraging PMF framework and bandit algorithms.…”

Section: Interactive Collaborative Filteringmentioning

confidence: 99%

“…Besides, user's information, e.g., demographic information (age and gender), is provided for each visit event, and represented as the user identi cation. is data set has been used for evaluating contextual bandit algorithms in [16], [15], [24], [29]. In our experiments, the visit events of the rst day are utilized for selecting proper parameters of ICTR model, while 2.0 million of the remaining are used for the validation.…”

Section: Datasets Descriptionmentioning

confidence: 99%

Online Interactive Collaborative Filtering Using Multi-Armed Bandit with Dependent Arms

Wang

Zeng

Zhou

et al. 2019

IEEE Trans. Knowl. Data Eng.

Self Cite

View full text Add to dashboard Cite

Online interactive recommender systems strive to promptly suggest to consumers appropriate items (e.g., movies, news articles) according to the current context including both the consumer and item content information. However, such context information is o en unavailable in practice for the recommendation, where only the users' interaction data on items can be utilized. Moreover, the lack of interaction records, especially for new users and items, worsens the performance of recommendation further. To address these issues, collaborative ltering (CF), one of the recommendation techniques relying on the interaction data only, as well as the online multi-armed bandit mechanisms, capable of achieving the balance between exploitation and exploration, are adopted in the online interactive recommendation se ings, by assuming independent items (i.e., arms). Nonetheless, the assumption rarely holds in reality, since the real-world items tend to be correlated with each other (e.g., two articles with similar topics).In this paper, we study online interactive collaborative ltering problems by considering the dependencies among items. We explicitly formulate the item dependencies as the clusters on arms, where the arms within a single cluster share the similar latent topics. In light of the topic modeling techniques, we come up with a generative model to generate the items from their underlying topics. Furthermore, an e cient online algorithm based on particle learning is developed for inferring both latent parameters and states of our model. Additionally, our inferred model can be naturally integrated with existing multi-armed selection strategies in the online interactive collaborating se ing. Empirical studies on two real-world applications, online recommendations of movies and news, demonstrate both the e ectiveness and e ciency of the proposed approach.

show abstract

“…To solve such contextual recommendation problems, algorithms based on contextual-bandits have been successfully deployed in a number of industrial level applications over the past decade, such as personalized recommender systems [17,27,28], advertisement personalization [6,29], and learning-to-rank [22]. Contextual-bandit algorithms are preferred if one needs to minimize the cumulative cost during online-learning because they aim to address the tradeoff between exploitation and exploration.…”

Section: Introductionmentioning

confidence: 99%

Policy Gradients for Contextual Recommendations

Pan

Cai

Tang

et al. 2019

The World Wide Web Conference

View full text Add to dashboard Cite

Decision making is a challenging task in online recommender systems. The decision maker often needs to choose a contextual item at each step from a set of candidates. Contextual bandit algorithms have been successfully deployed to such applications, for the tradeoff between exploration and exploitation and the state-of-art performance on minimizing online costs. However, the applicability of existing contextual bandit methods is limited by the over-simplified assumptions of the problem, such as assuming a simple form of the reward function or assuming a static environment where the states are not affected by previous actions.In this work, we put forward Policy Gradients for Contextual Recommendations (PGCR) to solve the problem without those unrealistic assumptions. It optimizes over a restricted class of policies where the marginal probability of choosing an item (in expectation of other items) has a simple closed form, and the gradient of the expected return over the policy in this class is in a succinct form. Moreover, PGCR leverages two useful heuristic techniques called Time-Dependent Greed and Actor-Dropout. The former ensures PGCR to be empirically greedy in the limit, and the latter addresses the trade-off between exploration and exploitation by using the policy network with Dropout as a Bayesian approximation.PGCR can solve the standard contextual bandits as well as its Markov Decision Process generalization. Therefore it can be applied to a wide range of realistic settings of recommendations, such as personalized advertising. We evaluate PGCR on toy datasets as well as a real-world dataset of personalized music recommendations. Experiments show that PGCR enables fast convergence and low regret, and outperforms both classic contextual-bandits and vanilla policy gradient methods.

show abstract

Personalized Recommendation via Parameter-Free Contextual Bandits

Cited by 61 publications

References 24 publications

Collaborative Filtering Bandits

Collaborative Filtering Bandits

Online Interactive Collaborative Filtering Using Multi-Armed Bandit with Dependent Arms

Policy Gradients for Contextual Recommendations

Contact Info

Product

Resources

About