Monte Carlo Estimates of Evaluation Metric Error and Bias

Abstract

Traditional offline evaluations of recommender systems apply metrics from machine learning and information retrieval in settings where their underlying assumptions no longer hold. This results in significant error and bias in measures of top-N recommendation performance, such as precision, recall, and nDCG. Several of the specific causes of these errors, including popularity bias and misclassified decoy items, are well-explored in the existing literature. In this paper we survey a range of work on identifying and addressing these problems, and report on our work in progress to simulate the recommender data generation and evaluation processes to quantify the extent of evaluation metric errors and assess their sensitivity to various assumptions.

Doing Data Right: How Lessons Learned Working with Conventional Data should Inform the Future of Synthetic Data for Recommender Systems

“…In [9], the authors proposed FaiRecSys, an algorithm that mitigates algorithmic bias by post-processing the recommendation matrix with minimum impact on the accuracy of recommendations provided to the end-users. Third, evaluation metric error and bias [33] simulates the recommender data generation and evaluation processes to quantify how erroneous current evaluation practices are. In [35], the authors proposed a simulation framework for measuring the impact of a recommender system under different types of user behavior.…”