Top-N-Rank: A Scalable List-wise Ranking Method for Recommender Systems

Abstract

We propose Top-N-Rank, a novel family of listwise Learning-to-Rank models for reliably recommending the N top-ranked items. The proposed models optimize a variant of the widely used discounted cumulative gain (DCG) objective function which differs from DCG in two important aspects: (i) It limits the evaluation of DCG only on the top N items in the ranked lists, thereby eliminating the impact of low-ranked items on the learned ranking function; and (ii) it incorporates weights that allow the model to leverage multiple types of implicit feedback with differing levels of reliability or trustworthiness. Because the resulting objective function is non-smooth and hence challenging to optimize, we consider two smooth approximations of the objective function, using the traditional sigmoid function and the rectified linear unit (ReLU). We propose a family of learning-to-rank algorithms (Top-N-Rank) that work with any smooth objective function. Then, a more efficient variant, Top-N-Rank.ReLU, is introduced, which effectively exploits the properties of ReLU function to reduce the computational complexity of Top-N-Rank from quadratic to linear in the average number of items rated by users. The results of our experiments using two widely used benchmarks, namely, the MovieLens data set and the Amazon Video Games data set demonstrate that: (i) The "top-N truncation" of the objective function substantially improves the ranking quality of the top N recommendations; (ii) using the ReLU for smoothing the objective function yields significant improvement in both ranking quality as well as runtime as compared to using the sigmoid; and (iii) Top-N-Rank.ReLU substantially outperforms the well-performing list-wise ranking methods in terms of ranking quality.

Differentiable Ranking Metric Using Relaxed Sorting for Top-K Recommendation

“…The weight Φ ui introduced in [5], [13] enables pairwise objectives to emphasize the loss of positive items at lower ranks; the value of Φ ui is chosen to be larger if the approximated rank is lower for a positive item i. Some choices of Φ ui are known to make the optimizing hinge loss closely related to maximizing discounted cumulative gain [18], [19].…”

New Insights into Metric Optimization for Ranking-based Recommendation

“…This method has been widely applied for optimizing [25], [39] and [40]. Rather than optimizing the whole list and taking items at the bottom into account, Liang et al [25] proposed Top-N-Rank, which focuses on the top ranked items and uses a listwise loss with a cutoff to directly optimize for @ . Despite this rich track record of attempts to learn a ranking by metric optimization, still insufficient is known about what metric to optimize for in order to obtain the best performance according to some evaluation metric.…”

“…Another popular method is to use a smooth function, such as a sigmoid or ReLU [32], to approximate the non-smooth indicator function. This method has been widely applied for optimizing 𝐷𝐶𝐺 [25], 𝐴𝑃 [39] and 𝑅𝑅 [40]. Rather than optimizing the whole list and taking items at the bottom into account, Liang et al [25] proposed Top-N-Rank, which focuses on the top ranked items and uses a listwise loss with a cutoff to directly optimize for 𝐷𝐶𝐺@𝑘.…”

“…This method has been widely applied for optimizing 𝐷𝐶𝐺 [25], 𝐴𝑃 [39] and 𝑅𝑅 [40]. Rather than optimizing the whole list and taking items at the bottom into account, Liang et al [25] proposed Top-N-Rank, which focuses on the top ranked items and uses a listwise loss with a cutoff to directly optimize for 𝐷𝐶𝐺@𝑘.…”

“…Therefore, it has been seen as an intuitive and logical choice to learn a ranking model for a recommender system by directly optimizing the metric used for evaluation [26]. Different ranking approaches have been designed along this line, aiming at achieving a better recommendation performance [25,[39][40][41].…”