Reenvisioning the comparison between Neural Collaborative Filtering and Matrix Factorization

Abstract

Collaborative filtering models based on matrix factorization and learned similarities using Artificial Neural Networks (ANNs) have gained significant attention in recent years. This is, in part, because ANNs have demonstrated very good results in a wide variety of recommendation tasks. However, the introduction of ANNs within the recommendation ecosystem has been recently questioned, raising several comparisons in terms of efficiency and effectiveness. One aspect most of these comparisons have in common is their focus on accuracy, neglecting other evaluation dimensions important for the recommendation, such as novelty, diversity, or accounting for biases. In this work, we replicate experiments from three different papers that compare Neural Collaborative Filtering (NCF) and Matrix Factorization (MF), to extend the analysis to other evaluation dimensions. First, our contribution shows that the experiments under analysis are entirely reproducible, and we extend the study including other accuracy metrics and two statistical hypothesis tests. Second, we investigated the Diversity and Novelty of the recommendations, showing that MF provides a better accuracy also on the long tail, although NCF provides a better item coverage and more diversified recommendation lists. Lastly, we discuss the bias effect generated by the tested methods. They show a relatively small bias, but other recommendation baselines, with competitive accuracy performance, consistently show to be less affected by this issue. This is the first work, to the best of our knowledge, where several complementary evaluation dimensions have been explored for an array of state-of-the-art algorithms covering recent adaptations of ANNs and MF. Hence, we aim to show the potential these techniques may have on beyond-accuracy evaluation while analyzing the effect on reproducibility these complementary dimensions may spark. The code to reproduce the experiments is publicly available on GitHub at https:// tny.sh/ Reenvisioning.

Revisiting the Performance of iALS on Item Recommendation Benchmarks

“…The benchmarks that we pick have been proposed and studied by multiple research groups [18,9,13,1,6] and other researchers have used them for evaluating newly proposed algorithms [26,14,19,27]. The poor iALS results have been established and reproduced by multiple groups [18,13,9,6,1] including a paper focused on reproducibility [1]. However, contrary to these results, we show that iALS can in fact generate high quality results on exactly the same benchmarks using exactly the same evaluation method.…”

“…The original comparison in [9] reports results for eALS [10] which is a coordinate descent variation of iALS. Follow up work [6] provided results for iALS which have been reproduced in [1]. The previously reported performance of eALS and iALS is poor and not competitive on both measures and both datasets.…”

“…We also investigate the performance of iALS on the sampled item recommendation benchmarks from [9] which have been used in multiple publications, including [23,18,6,1]. The benchmark contains two datasets, Pinterest [7] and an implicit version of Movielens 1M (ML1M) [8].…”

“…In this work, we revisit four wellstudied item recommendation benchmarks where poor prediction quality was reported for iALS. The benchmarks that we pick have been proposed and studied by multiple research groups [18,9,13,1,6] and other researchers have used them for evaluating newly proposed algorithms [26,14,19,27]. The poor iALS results have been established and reproduced by multiple groups [18,13,9,6,1] including a paper focused on reproducibility [1].…”

Learning edge importance in bipartite graph-based recommendations

“…G RAPH-BASED RP3Beta model [1] is a very strong baseline on multiple recommender systems datasets [2], [3], [4]. This relatively simple model outperformed other approaches on our published OLX Jobs Interactions dataset and is currently a state-of-the-art collaborative filtering recommender system at OLX.…”