Reproducibility of Experiments in Recommender Systems Evaluation

Abstract

Recommender systems evaluation is usually based on predictive accuracy metrics with better scores meaning recommendations of higher quality. However, the comparison of results is becoming increasingly difficult, since there are different recommendation frameworks and different settings in the design and implementation of the experiments. Furthermore, there might be minor differences on algorithm implementation among the different frameworks. In this paper, we compare well known recommendation algorithms, using the same dataset, metrics and overall settings, the results of which point to result differences across frameworks with the exact same settings. Hence, we propose the use of standards that should be followed as guidelines to ensure the replication of experiments and the reproducibility of the results.

A Guideline-Based Approach for Assisting with the Reproducibility of Experiments in Recommender Systems Evaluation

“…A major challenge in recommender systems' evaluation is that when a new recommendation algorithm is developed the concept of reproducibility is not considered in depth, thus making a study difficult to be reproduced in the future by other researchers. In our previous work we have shown that it is difficult to reproduce results using different evaluation libraries due to differences in algorithm and metric implementations, while highlighting that the results derived from well-known algorithms that can be found across libraries can be easily reproduced 1 . In recommender systems' evaluation to reproduce experiments it is recommended to follow a set of guidelines 1,2 .…”

“…During the last few years, research in recommender systems, both in academia and in industry, has led to numerous publications. The popularity of recommender systems' research has led to the increasingly important problem of reproducibility and replication of experiments during the evaluation of such systems 1,2 . The evaluation of recommendation algorithms is important for measuring the quality of the results and for making objective comparisons of algorithms.…”

“…All of these recommendation frameworks provide essentially the same portfolio of algorithms and evaluation metrics. Although, it might be difficult to reproduce a study from research papers the aforemention evaluation libraries provide all the main algorithms and evaluation metrics, while the results for those are very similar and comparable between them 1 . We focus on the reproducibility of new algorithms that will have to be reproduced in the future, despite of the framework that they will be based on.…”

“…State of the art work about the evaluation of recommender systems can be found in 3,9 . Research papers that propose new recommendation algorithms will typically describe the experimental setup, the dataset and the framework used; the term reproducibility is used to mean the replication and validation of the results by third parties 1,2 .…”

A Recommendation Model Based on Multi-Emotion Similarity in the Social Networks

“…A variety of recommendation systems have been proposed by researchers. The main recommendation systems include [2][3][4][5]: (1) Content-based recommendation systems, which will recommend the goods that a user is interested in based on their historical behaviors;…”