Evaluation of session-based recommendation algorithms

Abstract

Recommender systems help users find relevant items of interest, for example on e-commerce or media streaming sites. Most academic research is concerned with approaches that personalize the recommendations according to long-term user profiles. In many real-world applications, however, such long-term profiles often do not exist and recommendations therefore have to be made solely based on the observed behavior of a user during an ongoing session. Given the high practical relevance of the problem, an increased interest in this problem can be observed in recent years, leading to a number of proposals for session-based recommendation algorithms that typically aim to predict the user's immediate next actions.In this work, we present the results of an in-depth performance comparison of a number of such algorithms, using a variety of datasets and evaluation measures. Our comparison includes the most recent approaches based on recurrent neural networks like gru4rec, factorized Markov model approaches such as fism or fossil, as well as simpler methods based, e.g., on nearest neighbor schemes. Our experiments reveal that algorithms of this latter class, despite their sometimes almost trivial nature, often perform equally well or significantly better than today's more complex approaches based on deep neural networks. Our results therefore suggest that there is substantial room for improvement regarding the development of more sophisticated session-based recommendation algorithms. 1

Are we really making much progress? A worrying analysis of recent neural recommendation approaches

“…Therefore, progress is often claimed by comparing a complex neural model against another neural model, which is, however, not necessarily a strong baseline. Similar observations can be made for the area of session-based recommendation, where a recent method based on recurrent neural networks [16] is considered a competitive baseline, even though almost trivial methods are in most cases better [29,30].…”

“…The validation of the progress that is achieved through new methods against a set of baselines can be done in at least two ways. One is to evaluate all considered methods within the same defined environment, using the same datasets and the exact same evaluation procedure for all algorithms as done in [29]. While such an approach helps us obtain a picture of how different methods compare across datasets, the implemented evaluation procedure might be slightly different from the one used in the original papers.…”

Deep Learning-Based Sequential Recommender Systems: Concepts, Algorithms, and Evaluations

“…Repeat consumption refers to that an item is repeatedly appeared in a user's historical sequence, which is mostly ignored in sequential recommendation. Only RepeatNet proposed by Ren et al [64] 9 has ever considered the issue, and their results confirm that the consideration of repeat consumption in network design can improve the recommendation performance.…”

“…Frequent pattern mining. As we know, association rule [9] strives to use frequent pattern mining to mine frequent patterns with sufficient support and confidence. In sequential recommendation, a pattern is recognized only when the co-occurring items appear in the same order in different sequences, which is thus used to make recommendations.…”

Attentive Item2vec: Neural Attentive User Representations

“…The evaluation protocol is as follows: for each user = ( 1 ,… , ), we create − 1 examples {( 1: −1 , )} =2 where the task is to predict the target item for the query sub-user 1: −1 = ( 1 ,… , −1 ). We use two popular [1]- [3], [12], [16] Mean Reciprocal Rank at (MRR@ ): This measure is defined as the average of the reciprocal ranks [1], where the reciprocal rank is set to zero if the rank is above . Unlike HR@ , the MRR@ measure does consider the order of the recommendation list.…”