A Critical Analysis of Offline Evaluation Decisions Against Online Results: A Real-Time Recommendations Case Study

Gonçalves, Diogo; Marinho, Vanessa Queiroz; Magalhães, Ana Rita; Sá, João

doi:10.1007/978-3-030-94016-4_6

Cited by 3 publications

(2 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These strong results are consistent with our offline experiments. We believe this is because we train on diverse sources of interaction data from all products and premises which in turn helps to provide more personalized and relevant content and dampens feedback loops and selection biases [18,24].…”

Section: Online Resultsmentioning

confidence: 99%

“…This has two benefits, first, it significantly increases the training dataset size, and second, it dampens feedback loops, where the recommender system is trained on data from the same carousel on a previous day. Feedback loops can amplify biases such as popularity and presentation bias [1,4,9]. The same model is used to serve partial cold-start users by utilizing other types of interactions in the session as well as full cold-start users without interactions by utilizing "static" contextual information about the user.…”

Section: Model Architecturementioning

confidence: 99%

See 1 more Smart Citation

Reusable Self-Attention Recommender Systems in Fashion Industry Applications

Celikik¹,

Ramallo²,

Wasilewski³

2022

Proceedings of the 16th ACM Conference on Recommender Systems

View full text Add to dashboard Cite

A large number of empirical studies on applying self-attention models in the domain of recommender systems are based on offline evaluation and metrics computed on standardized datasets, without insights on how these models perform in real life scenarios. Moreover, many of them do not consider information such as item and customer metadata, although deep-learning recommenders live up to their full potential only when numerous features of heterogeneous types are included. Also, typically recommendation models are designed to serve well only a single use case, which increases modeling complexity and maintenance costs, and may lead to inconsistent customer experience. In this work, we present a reusable Attention-based Fashion Recommendation Algorithm (AFRA), that utilizes various interaction types with different fashion entities such as items (e.g., shirt), outfits and influencers, and their heterogeneous features. Moreover, we leverage temporal and contextual information to address both short and long-term customer preferences. We show its effectiveness on outfit recommendation use cases, in particular: 1) personalized ranked feed; 2) outfit recommendations by style; 3) similar item recommendation and 4) in-session recommendations inspired by most recent customer actions. We present both offline and online experimental results demonstrating substantial improvements in customer retention and engagement.

show abstract

Section: Online Resultsmentioning

confidence: 99%

Section: Model Architecturementioning

confidence: 99%