Contrasting Offline and Online Results when Evaluating Recommendation Algorithms

Rossetti, Marco; Stella, Fabio; Zanker, Markus

doi:10.1145/2959100.2959176

Cited by 65 publications

(35 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The hunt for better accuracy values dominates research activities in this area, even though it is not even clear if slightly higher accuracy values are relevant in terms of adding value for recommendation consumers or providers [20,22,52]. In fact, a number of research works exist that indicate that higher accuracy does not necessarily translate into betterreceived recommendations [4,9,13,31,37].…”

Section: Progress Assessmentmentioning

confidence: 99%

Are we really making much progress? A worrying analysis of recent neural recommendation approaches

Dacrema

Cremonesi

Jannach

2019

Proceedings of the 13th ACM Conference on Recommender Systems

413

166

View full text Add to dashboard Cite

Deep learning techniques have become the method of choice for researchers working on algorithmic aspects of recommender systems. With the strongly increased interest in machine learning in general, it has, as a result, become difficult to keep track of what represents the state-of-the-art at the moment, e.g., for top-n recommendation tasks. At the same time, several recent publications point out problems in today's research practice in applied machine learning, e.g., in terms of the reproducibility of the results or the choice of the baselines when proposing new models.In this work, we report the results of a systematic analysis of algorithmic proposals for top-n recommendation tasks. Specifically, we considered 18 algorithms that were presented at top-level research conferences in the last years. Only 7 of them could be reproduced with reasonable effort. For these methods, it however turned out that 6 of them can often be outperformed with comparably simple heuristic methods, e.g., based on nearest-neighbor or graph-based techniques. The remaining one clearly outperformed the baselines but did not consistently outperform a well-tuned nonneural linear ranking method. Overall, our work sheds light on a number of potential problems in today's machine learning scholarship and calls for improved scientific practices in this area.

show abstract

Section: Progress Assessmentmentioning

confidence: 99%

Are we really making much progress? A worrying analysis of recent neural recommendation approaches

Dacrema

Cremonesi

Jannach

2019

Proceedings of the 13th ACM Conference on Recommender Systems

413

166

View full text Add to dashboard Cite

show abstract

“…Next to offline evaluations possible through these data sets, it is worth noting that online evaluations still play an extensive role in evaluating a recommender system. User actual reactions might differ drastically from predictions made from offline data [78].…”

Section: Future Workmentioning

confidence: 98%

Recommender Systems for Health Informatics: State-of-the-Art and Future Perspectives

Valdez

Ziefle

Verbert

et al. 2016

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Recommender systems are a classical example for machine learning applications, however, they have not yet been used extensively in health informatics and medical scenarios. We argue that this is due to the specifics of benchmarking criteria in medical scenarios and the multitude of drastically differing end-user groups and the enormous contextcomplexity of the medical domain. Here both risk perceptions towards data security and privacy as well as trust in safe technical systems play a central and specific role, particularly in the clinical context. These aspects dominate acceptance of such systems. By using a Doctor-in-the-Loop approach some of these difficulties could be mitigated by combining both human expertise with computer efficiency. We provide a three-part research framework to access health recommender systems, suggesting to incorporate domain understanding, evaluation and specific methodology into the development process.

show abstract

“…This number can be variable. For example, in the MovieLens dataset 500, 1000, 1500 and 2000 items may be selected randomly [28,36].…”

Section: Related Workmentioning

confidence: 99%

A Fast Recommender System for Cold User Using Categorized Items

Jazayeriy

Mohammadi

Shamshirband

2018

MCA

View full text Add to dashboard Cite

Abstract:In recent years, recommender systems (RS) provide a considerable progress to users. RSs reduce the cost of a user's time in order to reach to desired results faster. The main issue of RSs is the presence of cold users which are less active and their preferences are more difficult to detect. The aim of this study is to provide a new way to improve recall and precision in recommender systems for cold users. According to the available categories of items, prioritization of the proposed items is improved and then presented to the cold user. The obtained results show that in addition to increased speed of processing, recall and precision have an acceptable improvement.

show abstract

Contrasting Offline and Online Results when Evaluating Recommendation Algorithms

Cited by 65 publications

References 12 publications

Are we really making much progress? A worrying analysis of recent neural recommendation approaches

Are we really making much progress? A worrying analysis of recent neural recommendation approaches

Recommender Systems for Health Informatics: State-of-the-Art and Future Perspectives

A Fast Recommender System for Cold User Using Categorized Items

Contact Info

Product

Resources

About