Offline Evaluation to Make Decisions About PlaylistRecommendation Algorithms

Gruson, Alois; Chandar, Praveen; Charbuillet, Christophe; McInerney, James O.; Hansen, Samantha; Tardieu, Damien; Carterette, Ben

doi:10.1145/3289600.3291027

Cited by 92 publications

(69 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Causal inference [15,28,37] estimates outcomes through the counterfactual reasoning. It has previously been used to evaluate recommendations in [5,7,8,16,25], which used IPS, SNIPS, and their extensions. These work evaluated purchases under recommendations, which is equivalent to the use of only the left terms in Eq.…”

Section: Related Work 41 Causal Inference For Recommendersmentioning

confidence: 99%

“…To intuitively understand the diference in the recommendation outputs between the accuracy-based optimization and uplift-based optimization, Table 6 shows the often-recommended items by RMF and ULRMF in the Dunnhumby dataset. While RMF tends to recommend popular items, ULRMF recommends items without an 3JARRED FRUIT(889) POTATO CHIPS 7TEA UNSWEETENED(833) SFT DRNK 2LITER BTL (6) NUTS OTHER(829) BEERALEMALT LIQUORS (11) INFANT FORMULA TODDLER(863) 100% PURE JUICE ORANGE (8) DECOR BULBS(687) TOILET TISSUE (10) FLUID MILK WHITE ONLY(1) TORTILLA/NACHO CHIPS (15) BEEF STEW(638) emphasis on popular ones 14 . Often-recommended items by ULRMF include those that might induce impulse purchases such as pasta sauce and heat-and-serve meals.…”

Section: Trends Of the Recommended Items (Rq3)mentioning

confidence: 99%

See 1 more Smart Citation

Uplift-based evaluation and optimization of recommenders

Sato

Singh

Takemori

et al. 2019

Proceedings of the 13th ACM Conference on Recommender Systems

View full text Add to dashboard Cite

Recommender systems aim to increase user actions such as clicks and purchases. Typical evaluations of recommenders regard the purchase of a recommended item as a success. However, the item may have been purchased even without the recommendation. An uplift is defned as an increase in user actions caused by recommendations. Situations with and without a recommendation cannot both be observed for a specifc user-item pair at a given time instance, making uplift-based evaluation and optimization challenging. This paper proposes new evaluation metrics and optimization methods for the uplift in a recommender system. We apply a causal inference framework to estimate the average uplift for the ofine evaluation of recommenders. Our evaluation protocol leverages both purchase and recommendation logs under a currently deployed recommender system, to simulate the cases both with and without recommendations. This enables the ofine evaluation of the uplift for newly generated recommendation lists. For optimization, we need to defne positive and negative samples that are specifc to an uplift-based approach. For this purpose, we deduce four classes of items by observing purchase and recommendation logs. We derive the relative priorities among these four classes in terms of the uplift and use them to construct both pointwise and pairwise sampling methods for uplift optimization. Through dedicated experiments with three public datasets, we demonstrate the efectiveness of our optimization methods in improving the uplift. CCS CONCEPTS • Information systems → Recommender systems; • Computing methodologies → Learning from implicit feedback.

show abstract

Section: Related Work 41 Causal Inference For Recommendersmentioning

confidence: 99%

Section: Trends Of the Recommended Items (Rq3)mentioning

confidence: 99%

Uplift-based evaluation and optimization of recommenders

Sato

Singh

Takemori

et al. 2019

Proceedings of the 13th ACM Conference on Recommender Systems

View full text Add to dashboard Cite

show abstract

“…This method manages to effectively exploit information and to quickly rank playlists, which is an interesting result, as fully-personalized contextual models were actually the only ones able to learn the exact display-to-stream probabilities (see generative process in Section 3.2), and as both frameworks have comparable numbers of parameters (K × Q vs K × D). While fully-personalized methods have been the focus of previous works on carousel recommendation [12,32], our experiments emphasize the empirical benefit of semi-personalization via user clustering that, assuming good underlying clusters, might appear as a suitable alternative for such large-scale real-world applications.…”

Section: Algorithmsmentioning

confidence: 77%

“…. This corresponds to the contextual bandit setting [1,7,28], a popular learning paradigm for online recommender systems [12,28,29,32,35,41,43,47,48]. Strategies to learn weight vectors are detailed e.g.…”

Section: Contextualmentioning

confidence: 99%

Carousel Personalization in Music Streaming Apps with Contextual Bandits

Bendada¹,

Salha²,

Bontempelli³

2020

Fourteenth ACM Conference on Recommender Systems

View full text Add to dashboard Cite

Media services providers, such as music streaming platforms, frequently leverage swipeable carousels to recommend personalized content to their users. However, selecting the most relevant items (albums, artists, playlists...) to display in these carousels is a challenging task, as items are numerous and as users have different preferences. In this paper, we model carousel personalization as a contextual multi-armed bandit problem with multiple plays, cascade-based updates and delayed batch feedback. We empirically show the effectiveness of our framework at capturing characteristics of real-world carousels by addressing a large-scale playlist recommendation task on a global music streaming mobile app. Along with this paper, we publicly release industrial data from our experiments, as well as an open-source environment to simulate comparable carousel personalization learning problems.

show abstract

“…CTR is especially significant in applications involving online advertisement/purchase where the recommendations are made based on CTR × utility, and utility is the revenue the click generates. Better CTR prediction is a crucial aspect in determining the revenue of the system [9,11].…”

Section: Introductionmentioning

confidence: 99%

AutoFeature: Searching for Feature Interactions and Their Architectures for Click-through Rate Prediction

Khawar

Tang

et al. 2020

Proceedings of the 29th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

Click-through rate prediction is an important task in commercial recommender systems and it aims to predict the probability of a user clicking on an item. The event of a user clicking on an item is accompanied by several user and item features. As modelling the feature interactions effectively can lead to better predictions, it has been the focus of many recent approaches including deep learning-based models. However, the existing approaches either (i) model all possible feature interactions for a given order, or (ii) manually select which feature interactions to model. Besides, they use the same network structure or function to model all the feature interactions while ignoring the difference of complexity among them. To address these issues, we propose a neural architecture search based approach called AutoFeature that automatically finds essential feature interactions and selects an appropriate structure to model each of these interactions. Specifically, we first define a flexible architecture search space for the CTR prediction task which covers many popular designs such as PIN, PNN and DeepFM, etc., and enables higher-order interactions. Then we propose an efficient neural architecture search algorithm that recursively refines the search space by partitioning it into several subspaces and samples from higher quality ones. Extensive experiments on multiple CTR prediction benchmarks show the superiority of our AutoFeature over the state-of-the-art baselines. In addition, our experiments show that the learned architectures use fewer flops/parameters and hence can efficiently incorporate higher-order feature interactions. This further boosts the performance. Finally, we show that AutoFeature can find meaningful feature interactions.

show abstract

Offline Evaluation to Make Decisions About PlaylistRecommendation Algorithms

Cited by 92 publications

References 36 publications

Uplift-based evaluation and optimization of recommenders

Uplift-based evaluation and optimization of recommenders

Carousel Personalization in Music Streaming Apps with Contextual Bandits

AutoFeature: Searching for Feature Interactions and Their Architectures for Click-through Rate Prediction

Contact Info

Product

Resources

About