Elliot: A Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation

Anelli, Vito Walter; Bellogín, Alejandro; Ferrara, Antonio; Malitesta, Daniele; Merra, Felice Antonio; Pomo, Claudio; Donini, Francesco M.; Noia, Tommaso Di

doi:10.1145/3404835.3463245

Cited by 49 publications

(12 citation statements)

References 92 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fortunately, the emergence of works such as the previously mentioned [23] and [24] have raised much concern, and various proposals for comprehensive recom-mendation frameworks are starting to be proposed. Many of these issues are being addressed by the excellent works of [192], [284], [315], [317].…”

Section: Summary and Discussionmentioning

confidence: 99%

“…Other properties related to user experience such as non-intrusiveness, trustworthiness and other matters related to privacy have also been discussed with great interest [4]. Finally, much could be said about metrics related to fairness and novelty, partly related to the matters of bias and diversity we discussed before, and that are seeing more and more interest in recent research [284].…”

Section: E Other Issues and Extensionsmentioning

confidence: 99%

“…Keeping such considerations in mind, it comes as no surprise that many researchers use sampling strategies to speed up the evaluation process; since datasets are usually very sparse, some approaches sample a subset of unobserved items for every user. However, several studies have pointed out the difficulty of obtaining reliable performance results using metrics with sampling strategies [281]- [284]. Specifically, [281] demonstrates how sampled metrics are, in fact, not good indicators of the model performance when compared to the same global, non-sampled metrics.…”

Section: Sampled Metricsmentioning

confidence: 99%

“…We occasionally found it not possible to determine whether reported metrics had been computed using comparable data splits or whether they relied on sampling strategies, which would prevent direct comparison. As pointed out in [284], [312], the selected strategy to split data in training and testing set (and possibly to generate sessions) from user's historical behavior can have a considerable impact on the measured performance. Hence, methods using different data splits, even when created from the same datasets, cannot always be compared reliably without repeating tests on the same preprocessed data.…”

Section: Foreword: the Issue Of Reproducibilitymentioning

confidence: 99%

“…Metrics and datasets seem to be conveniently chosen and paired with inadvertently weak baselines, giving the impression of improving a few performance metrics, despite several works warning there may not be a direct correlation between accuracy and improved recommendations [23], [313], [314]. While we understand how many of these works are custom-tailored to solve domain-specific problems and may well be worthy of attention, our goal is to show how slight variations in the problem formulation, data processing, and metrics of choice create a very fragmented landscape that lacks established benchmarking strategies of reference [25], [284].…”

Section: Foreword: the Issue Of Reproducibilitymentioning

confidence: 99%

See 4 more Smart Citations

Recommendation Systems: An Insight Into Current Development and Future Research Challenges

et al. 2022

View full text Add to dashboard Cite

Research on recommendation systems is swiftly producing an abundance of novel methods, constantly challenging the current state-of-the-art. Inspired by advancements in many related fields, like Natural Language Processing and Computer Vision, many hybrid approaches based on deep learning are being proposed, making solid improvements over traditional methods. On the downside, this flurry of research activity, often focused on improving over a small number of baselines, makes it hard to identify reference methods and standardized evaluation protocols. Furthermore, the traditional categorization of recommendation systems into content-based, collaborative filtering and hybrid systems lacks the informativeness it once had. With this work, we provide a gentle introduction to recommendation systems, describing the task they are designed to solve and the challenges faced in research. Building on previous work, an extension to the standard taxonomy is presented, to better reflect the latest research trends, including the diverse use of content and temporal information. To ease the approach toward the technical methodologies recently proposed in this field, we review several representative methods selected primarily from top conferences and systematically describe their goals and novelty. We formalize the main evaluation metrics adopted by researchers and identify the most commonly used benchmarks. Lastly, we discuss issues in current research practices by analyzing experimental results reported on three popular datasets.

show abstract