Improving accountability in recommender systems research through reproducibility

Abstract

Reproducibility is a key requirement for scientific progress. It allows the reproduction of the works of others, and, as a consequence, to fully trust the reported claims and results. In this work, we argue that, by facilitating reproducibility of recommender systems experimentation, we indirectly address the issues of accountability and transparency in recommender systems research from the perspectives of practitioners, designers, and engineers aiming to assess the capabilities of published research works. These issues have become increasingly prevalent in recent literature. Reasons for this include societal movements around intelligent systems and artificial intelligence striving toward fair and objective use of human behavioral data (as in Machine Learning, Information Retrieval, or Human–Computer Interaction). Society has grown to expect explanations and transparency standards regarding the underlying algorithms making automated decisions for and around us. This work surveys existing definitions of these concepts and proposes a coherent terminology for recommender systems research, with the goal to connect reproducibility to accountability. We achieve this by introducing several guidelines and steps that lead to reproducible and, hence, accountable experimental workflows and research. We additionally analyze several instantiations of recommender system implementations available in the literature and discuss the extent to which they fit in the introduced framework. With this work, we aim to shed light on this important problem and facilitate progress in the field by increasing the accountability of research.

Multi-Objective Entropy FCSVD: Contextual Spectrum Analysis for Prototyping Recommender Systems

“…The results of the Cold Start Challenge study in the fact that new users or new items entering the system adapt to changes in existing users or items [Kulkarni et al,2020]. Investigating the relationship between contextual information and how to use them in the recommendation process with MF methods [Bellogín et al,2021] and on the other hand applying the similarity measure in the preferences of the user [Chen et al,2021], demographic data [Natarajan et al,2020], social data [Boratto et al,2021,Ortiz et al,2022, etc. common ways to solve cold start.…”

“…A two-level CTLSVD matrix based on refining the properties of items and users based on the Time contextual information due to the slope of the descending gradient to reduce the challenge of sparse data [Fernández-García et al,2022]. Mainly similarity measure of contextual information is used to reduce sparse data [Bellogín et al,2021]. The use of contextual weighting overlaps measure is used to find the amount of contextual similarity.…”

“…Similarity criteria, such as Pearson correlation coefficient, cosine similarity, and their different types are often used to calculate the similarity of a pair of users or between pairs of items, but to reduce cold start using similar criteria, new combinations similar to NHSM and PSS can be used. Have a better performance in suggestions to users [Bellogín et al,2021].…”

Fairness in Recommender Systems: Research Landscape and Future Directions

“…Reproducibility. The lack of reproducibility can be a major barrier to achieve progress in AI [134], and recent studies indicate that limited reproducibility is a substantial issue also in recommender systems research [135,136]. Figure 11 shows for how many of the studied technical papers, artifacts were shared to ensure reproducibility of the reported experiments.…”

Elliot: a Comprehensive and Rigorous Framework for Reproducible Recommender Systems Evaluation

“…Some researchers argue that to enhance reproducibility, and to facilitate fair comparisons between different works (either frameworks, research papers, or published artifacts), at least the following four stages must be identified within the evaluation protocol [79]: data splitting, item recommendations, candidate item generation, and performance measurement. In a recent work [11], these stages have been completed with dataset collection and statistical testing. Some of these stages can be further categorized, such as performance measurement, depending on the performance dimension to be analyzed (e.g., ranking vs error, accuracy vs diversity, and so on).…”