Deep learning for recommender systems: A Netflix case study

Steck, Harald; Baltrunas, Linas; Elahi, Ehtsham; Liang, Dawen; Raimond, Yves; Basilico, Justin

doi:10.1609/aimag.v42i3.18140

Cited by 62 publications

(31 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Performance of neural methods: The two neural methods considered here, NeuMF and MultiVAE, only led to medium performance on these datasets. While MultiVAE performed very well in an earlier comparison with traditional methods [11], we may assume that the modest size of the datasets might limit the power of this method in our experiment to a certain extent, see also the report on the use of deep learning methods at Netflix [34] or the discussions in Jannach et al [16].…”

Section: Accuracy Resultsmentioning

confidence: 91%

Top-N Recommendation Algorithms: A Quest for the State-of-the-Art

Anelli,

Bellogín,

Di Noia

et al. 2022

Preprint

View full text Add to dashboard Cite

Research on recommender systems algorithms, like other areas of applied machine learning, is largely dominated by efforts to improve the state-of-the-art, typically in terms of accuracy measures. Several recent research works however indicate that the reported improvements over the years sometimes "don't add up", and that methods that were published several years ago often outperform the latest models when evaluated independently. Different factors contribute to this phenomenon, including that some researchers probably often only fine-tune their own models but not the baselines.In this paper, we report the outcomes of an in-depth, systematic, and reproducible comparison of ten collaborative filtering algorithmscovering both traditional and neural models-on several common performance measures on three datasets which are frequently used for evaluation in the recent literature. Our results show that there is no consistent winner across datasets and metrics for the examined top-n recommendation task. Moreover, we find that for none of the accuracy measurements any of the considered neural models led to the best performance. Regarding the performance ranking of algorithms across the measurements, we found that linear models, nearestneighbor methods, and traditional matrix factorization consistently perform well for the evaluated modest-sized, but commonly-used datasets. Our work shall therefore serve as a guideline for researchers regarding existing baselines to consider in future performance comparisons. Moreover, by providing a set of fine-tuned baseline models for different datasets, we hope that our work helps to establish a common understanding of the state-of-the-art for top-n recommendation tasks.

show abstract

Section: Accuracy Resultsmentioning

confidence: 91%

Top-N Recommendation Algorithms: A Quest for the State-of-the-Art

Anelli,

Bellogín,

Di Noia

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Given this continued research interest, the use of latest deep learning technology also in industry, for example (Steck et al. 2021), and the high quality of many deployed systems, one might think that the recommendation problem is almost solved. However, looking closer at today's published research on recommender systems, we find that the community seems to mainly focus on a rather narrow part of the overall problem setting.…”

Section: Recommender Systems – a Success Story Mostlymentioning

confidence: 99%

“…Unfortunately, similar problems exist for commonly used accuracy metrics. There are a number of reports signifying that improved offline accuracy does not translate to improvements in terms of key performance indicators of a deployed application, see also for a discussion of experiences at Netflix (Gomez-Uribe and Hunt 2015; Steck et al 2021).…”

Section: Improving Offline Evaluationmentioning

confidence: 99%

Recommender systems: Trends and frontiers

Jannach

Ricci

et al. 2022

AI Magazine

View full text Add to dashboard Cite

Recommender systems (RSs), as used by Netflix, YouTube, or Amazon, are one of the most compelling success stories of AI. Enduring research activity in this area has led to a continuous improvement of recommendation techniques over the years, and today's RSs are indeed often capable to make astonishingly good suggestions. With countless papers being published on the topic each year, one might think the recommendation problem is almost solved. In reality, however, the large majority of published works focuses on algorithmic improvements and relies on data-based evaluation procedures which may sometimes tell us little regarding the effects new algorithms will have in practice. This special issue contains a set of papers which address some of the open challenges and frontiers in RSs research: (i) building interactive and conversational solutions, (ii) understanding recommender systems as socio-technical systems with longitudinal dynamics, (iii) avoiding abstraction traps, and (iv) finding better ways of assessing the impact and value of recommender systems without field tests. RECOMMENDER SYSTEMS -A SUCCESS STORY, MOSTLYPersonalized suggestions for items to buy, news to read or movies to watch are nowadays ubiquitous on the web. Recommender systems are software solutions that generate these suggestions, commonly with the help of statistical models and machine learning techniques. Given their widespread use in practice, their often astonishingly good recommendations, and their proven value for consumers and providers, it is no surprise that research on recommender systems is flourishing. Today, we are witnessing a constantly growing interest in the topic both in academia and industry, with countless papers being published every year.Given this continued research interest, the use of latest deep learning technology also in industry, for example (Steck et al. 2021), and the high quality of many deployedThis is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

show abstract

“…For all these reasons, the significance of such an offline evaluation approach has been criticised. Ultimately, offline evaluations can lead to results that do not necessarily correlate with the performance of the RS measured in an online evaluation (Steck et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

Choice Models and Recommender Systems Effects on Users’ Choices

Hazrati

Ricci

2022

Preprint

View full text Add to dashboard Cite

Nowadays, the users of a web platform, such as a video-on-demand service or an eCommerce site, are routinely using the platform's Recommender System (RS) when choosing which item to consume or buy (e.g., movies or books). It is therefore important to understand how exposure to recommendations can influence the users' choices, particularly the quality and distribution of the chosen items. However, users, even in the presence of the same RS, may show diverse and even atypical choice behaviours, which are independent of the RS; they may have the preference for choosing more popular or recent items. The effect of these behaviours on the collective evolution of the choices and the performance of the RS is not well understood yet. In fact, in previous analyses, the users were supposed to only choose among the top recommendations, without any further discrimination. Hence, we first perform a correlation analysis, in some choices data sets, revealing that three kinds of choice behaviours, namely the tendency to choose popular, recent, and highly rated items, is actually observable in large percentages of the users. Then we investigate how these choice behaviours, implemented as algorithmic Choice Models (Popularity-CM, Age-CM and Rating-CM), can influence the overall choice distribution and the performance of the RS. With the aim of understanding such relationships and consequences, we have designed a simulation framework where the considered choice models (CMs) are adopted to simulate users' choices when they are exposed to recommendations from alternative RSs. We found that, 1) the choices' distribution of a user population is significantly influenced not only by the RS, but also substantially by the prevalent choice model of the population, 2) some biases of the RS are translated to the choices whatever is the CM, and 3) some important biases of the RS, in how they influence users' choice distribution, depend also on the choice model that the users adopt. The study contributes to the start of a new line of research where the impact of recommendation technologies can be studied with respect to alternative decision making approaches, which are actually followed by real users. Additionally, the simulation approach can help other researchers and practitioners to investigate the effect of deploying an RS when a certain CM is identified in a population of users.

show abstract

Deep learning for recommender systems: A Netflix case study

Cited by 62 publications

References 61 publications

Top-N Recommendation Algorithms: A Quest for the State-of-the-Art

Top-N Recommendation Algorithms: A Quest for the State-of-the-Art

Recommender systems: Trends and frontiers

Choice Models and Recommender Systems Effects on Users’ Choices

Contact Info

Product

Resources

About