Leveraging post-click feedback for content recommendations

Wen, Hongyi; Yang, Longqi; Estrin, Deborah

doi:10.1145/3298689.3347037

Cited by 36 publications

(36 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An interaction is identified as false-positive or true-positive one according to the explicit feedback. For instance, a purchase is false-positive if the following rating score ( [1,5]) < 3. Although such explicit feedback is sparse, the scale is sufficient to conduct a pilot experiment and construct a clean testing set.…”

Section: Study On False-positive Feedbackmentioning

confidence: 99%

“…As the clue to user choices, implicit feedback (e.g., click and purchase) are typically the default choice to train a recommender model due to their large volume. However, prior work [2], [4], [5] points out the gap between implicit feedback and the actual user satisfaction due to the existence of noisy interactions (a.k.a. false-positive interactions) where the users dislike the interacted items.…”

Section: Introductionmentioning

confidence: 99%

“…This is because implicit interactions are easily affected by the first impression of users and factors [6] like caption bias [7] and position bias [8]. Moreover, existing studies [5] have validated the detrimental effect of such noisy interactions on user experience. However, little work on recommendation has considered the noisy nature of implicit feedback.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Learning Robust Recommender from Noisy Implicit Feedback

Wang¹,

Feng²,

He³

et al. 2021

Preprint

View full text Add to dashboard Cite

The ubiquity of implicit feedback makes it indispensable for building recommender systems. However, it does not actually reflect the actual satisfaction of users. For example, in E-commerce, a large portion of clicks do not translate to purchases, and many purchases end up with negative reviews. As such, it is of importance to account for the inevitable noises in implicit feedback. However, little work on recommendation has taken the noisy nature of implicit feedback into consideration. In this work, we explore the central theme of denoising implicit feedback for recommender learning, including training and inference. By observing the process of normal recommender training, we find that noisy feedback typically has large loss values in the early stages. Inspired by this observation, we propose a new training strategy named Adaptive Denoising Training (ADT), which adaptively prunes the noisy interactions by two paradigms (i.e., Truncated Loss and Reweighted Loss). Furthermore, we consider extra feedback (e.g., rating) as auxiliary signal and propose three strategies to incorporate extra feedback into ADT: finetuning, warm-up training, and colliding inference. We instantiate the two paradigms on the widely used binary cross-entropy loss and test them on three representative recommender models. Extensive experiments on three benchmarks demonstrate that ADT significantly improves the quality of recommendation over normal training without using extra feedback. Besides, the proposed three strategies for using extra feedback largely enhance the denoising ability of ADT.

show abstract

Section: Study On False-positive Feedbackmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learning Robust Recommender from Noisy Implicit Feedback

Wang¹,

Feng²,

He³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Modelling and understanding skipping behaviour in music listening sessions arguably plays a crucial role in understanding user behaviour in modern streaming services. For instance, the skipping signal has already been used as a measure in heuristic-based playlist generation systems [9,25], user satisfaction [16,28], relevance [17], and as counterfactual estimators in Recommender Systems [22].…”

Section: Introductionmentioning

confidence: 99%

On Skipping Behaviour Types in Music Streaming Sessions

Meggetto

Revie

Levine

et al. 2021

Proceedings of the 30th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

The ability to skip songs is a core feature in modern online streaming services. Its introduction has led to a new music listening paradigm and has changed the way users interact with the underlying services. Thus, understanding their skipping activity during listening sessions has acquired considerable importance. This is because such implicit feedback signal can be considered a measure of users' satisfaction (dissatisfaction or lack of interest), affecting their engagement with the platforms. Prior work has mainly focused on analysing the skipping activity at an individual song level. In this work, we investigate different behaviours during entire listening sessions with regards to the users' session-based skipping activity. To this end, we propose a data transformation and clustering-based approach to identify and categorise skipping types. Experimental results on the real-world music streaming dataset (Spotify) indicate four main types of session skipping behaviour. A subsequent analysis of short, medium, and long listening sessions demonstrate that these session skipping types are consistent across sessions of varying length. Furthermore, we discuss their distributional differences under various listening context information, i.e. day types (i.e. weekday and weekend), times of the day, and playlist types. CCS CONCEPTS• Information systems → Recommender systems.

show abstract

“…These metrics assume that the implicit user feedback is indicative of how much they value their user experience. This assumption has been challenged recently [6,9,17,24]. In order to better understand and improve user experience on the platform, recommender systems have started to rely more on surveys in which users are explicitly asked to rate their experience on the platform, or specific items they have recently consumed [7,8,13,16].…”

Section: Introductionmentioning

confidence: 99%

Deconfounding User Satisfaction Estimation from Response Rate Bias

Christakopoulou

Traverse

Potter

et al. 2020

Fourteenth ACM Conference on Recommender Systems

View full text Add to dashboard Cite

Improving user satisfaction is at the forefront of industrial recommender systems. While significant progress has been made by utilizing logged implicit data of user-item interactions (i.e., clicks, dwell/watch time, and other user engagement signals), there has been a recent surge of interest in measuring and modeling user satisfaction, as provided by orthogonal data sources. Such data sources typically originate from responses to user satisfaction surveys, which explicitly ask users to rate their experience with the system and/or specific items they have consumed in the recent past. This data can be valuable for measuring and modeling the degree to which a user has had a satisfactory experience on the recommendation platform, since what users do (engagement) does not always align with what users say they want (satisfaction as measured by surveys). We focus on a large-scale industrial system trained on user survey responses to predict user satisfaction. The predictions of the satisfaction model for each user-item pair, combined with the predictions of the other models (e.g., engagement-focused ones), are fed into the ranking component of a real-world recommender system in deciding items to present to the user. It is therefore imperative that the satisfaction model does an equally good job on imputing user satisfaction across slices of users and items, as it would directly impact which items a user is exposed to. However, the data used for training satisfaction models is biased in that users are more likely to respond to a survey when they will respond that they are more satisfied. When the satisfaction survey responses in slices of data with high response rate follow a different distribution than those with low response rate, response rate becomes a confounding factor for user satisfaction estimation. We find positive correlation between response rate and ratings in a large-scale survey dataset collected in our case study. To address this inherent response rate bias in the satisfaction data, we propose an inverse propensity weighting approach within a multi-task learning framework. We extend a simple feed-forward neural network architecture predicting user satisfaction to a shared-bottom multi-task learning architecture with two tasks: the user satisfaction estimation task, and the response rate estimation task. We Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

show abstract

Leveraging post-click feedback for content recommendations

Cited by 36 publications

References 27 publications

Learning Robust Recommender from Noisy Implicit Feedback

Learning Robust Recommender from Noisy Implicit Feedback

On Skipping Behaviour Types in Music Streaming Sessions

Deconfounding User Satisfaction Estimation from Response Rate Bias

Contact Info

Product

Resources

About