Are we really making much progress? A worrying analysis of recent neural recommendation approaches

Dacrema, Maurizio Ferrari; Cremonesi, Paolo; Jannach, Dietmar

doi:10.1145/3298689.3347058

Cited by 413 publications

(217 citation statements)

References 50 publications

Supporting

Mentioning

166

Contrasting

Unclassified

Order By: Relevance

“…We do not use Neural Collaborative Filtering [11] or Bayesian Personalized Ranking (BPR) [17] as baselines, as their accuracies were found to be so low on these data-sets that no results were reported in [14], see also [5]. We compare these baselines to the following variants of the ADMM-approach outlined in Section 3:…”

Section: Methodsmentioning

confidence: 99%

Admm Slim

Steck

Dimakopoulou

Riabov

et al. 2020

Proceedings of the 13th International Conference on Web Search and Data Mining

View full text Add to dashboard Cite

The Sparse Linear Method (Slim) [15] is a well-established approach for top-N recommendations. This article proposes several improvements that are enabled by the Alternating Directions Method of Multipliers (ADMM) [3,8,9], a well-known optimization method with many application areas. First, we show that optimizing the original Slim-objective by ADMM results in an approach where the training time is independent of the number of users in the training data, and hence trivially scales to large numbers of users. Second, the flexibility of ADMM allows us to switch on and off the various constraints and regularization terms in the original Slim-objective, in order to empirically assess their contributions to ranking accuracy on given data. Third, we also propose two extensions to the original Slim training-objective in order to improve recommendation accuracy further without increasing the computational cost. In our experiments on three well-known data-sets, we first compare to the original Slim-implementation and find that not only ADMM reduces training time considerably, but also achieves an improvement in recommendation accuracy due to better optimization. We then compare to various state-of-the-art approaches and observe up to 25% improvement in recommendation accuracy in our experiments. Finally, we evaluate the importance of sparsity and the non-negativity constraint in the original Slim-objective with subsampling experiments that simulate scenarios of cold-starting and large catalog sizes compared to relatively small user base, which often occur in practice.

show abstract

Section: Methodsmentioning

confidence: 99%

Admm Slim

Steck

Dimakopoulou

Riabov

et al. 2020

Proceedings of the 13th International Conference on Web Search and Data Mining

View full text Add to dashboard Cite

show abstract

“…It is worthwhile noting that deep learning (DL) models for personalized recommendation have emerged in recent years. We acknowledge that whether DL models truly improve the recommendation performance is controversial (Dacrema et al, 2019), but this problem is out of the scope of this survey. In this survey, we will focus on the problem that the black-box nature of deep models brings difficulty in model explainability.…”

Section: A Historical Overviewmentioning

confidence: 99%

Explainable Recommendation: A Survey and New Perspectives

Zhang

Chen

2020

FNT in Information Retrieval

521

241

View full text Add to dashboard Cite

Explainable recommendation attempts to develop models that generate not only high-quality recommendations but also intuitive explanations. The explanations may either be post-hoc or directly come from an explainable model (also called interpretable or transparent model in some contexts). Explainable recommendation tries to address the problem of why: by providing explanations to users or system designers, it helps humans to understand why certain items are recommended by the algorithm, where the human can either be users or system designers. Explainable recommendation helps to improve the transparency, persuasiveness, effectiveness, trustworthiness, and satisfaction of recommendation systems. It also facilitates system designers for better system debugging. In recent years, a large number of explainable recommendation approaches -especially model-based methods -have been proposed and applied in real-world systems.In this survey, we provide a comprehensive review for the explainable recommendation research. We first highlight the position of explainable recommendation in recommender system research by categorizing recommendation problems into the 5W, i.e., what, when, who, where, and why. We then conduct a comprehensive survey of explainable recommendation on three perspectives: 1) We provide a chronological research timeline of explainable recommendation, including user study approaches in the early years and more recent model-based approaches. 2) We provide a two-dimensional taxonomy to classify existing explainable recommendation research: one dimension is the information source (or display style) of the explanations, and the other dimension is the algorithmic mechanism to generate explainable recommendations. 3) We summarize how explainable recommendation applies to different recommendation tasks, such as product recommendation, social recommendation, and POI recommendation.We also devote a section to discuss the explanation perspectives in broader IR and AI/ML research. We end the survey by discussing potential future directions to promote the explainable recommendation research area and beyond.Explainable recommendation refers to personalized recommendation algorithms that address the problem of why -they not only provide users or system designers with recommendation results, but also explanations to clarify why such items are recommended. In this way, it helps to improve the transparency, persuasiveness, effectiveness, trustworthiness, and user satisfaction of the recommendation systems. It also facilitates system designers to diagnose, debug, and refine the recommendation algorithm.To highlight the position of explainable recommendation in the recommender system research area, we classify personalized recommendation with a broad conceptual taxonomy. Specifically, personalized recommendation research can be classified into the 5W problemswhen, where, who, what, and why, corresponding to time-aware recommendation (when), location-based recommendation (where), social recommendation (who), application-aware re...

show abstract

“…This is not ideal, as usually the approach's implementation as well as additional information is needed to facilitate reproducing the experiment and results. For example, in [6], it is reported that only 7 out of 18 major neural recommendation approaches published at top tier conferences were reproducible, as only limited information was shared by the authors. Following the PRIMAD model [4,7] would resemble a large step towards reproducibility.…”

Section: Requirementsmentioning

confidence: 99%

“…BioASQ. The challenge on biomedical semantic indexing and question answering (BioASQ) [27] aims to improve the indexing process of PubMed 6 articles with the annotation of terms from the Medical Subject Headings (MeSH) 7 . In 2013, the BioASQ challenge was initiated in cooperation with the National Library of Medicine (NLM).…”

Section: Evaluation Infrastructuresmentioning

confidence: 99%

Evaluation Infrastructures for Academic Shared Tasks

et al. 2020

View full text Add to dashboard Cite

Academic search systems aid users in finding information covering specific topics of scientific interest and have evolved from early catalog-based library systems to modern web-scale systems. However, evaluating the performance of the underlying retrieval approaches remains a challenge. An increasing amount of requirements for producing accurate retrieval results have to be considered, e.g., close integration of the system's users. Due to these requirements, small to mid-size academic search systems cannot evaluate their retrieval system in-house. Evaluation infrastructures for shared tasks alleviate this situation. They allow researchers to experiment with retrieval approaches in specific search and recommendation scenarios without building their own infrastructure. In this paper, we elaborate on the benefits and shortcomings of four state-of-the-art evaluation infrastructures on search and recommendation tasks concerning the following requirements: support for online and offline evaluations, domain specificity of shared tasks, and reproducibility of experiments and results. In addition, we introduce an evaluation infrastructure concept design aiming at reducing the shortcomings in shared tasks for search and recommender systems.

show abstract

Are we really making much progress? A worrying analysis of recent neural recommendation approaches

Cited by 413 publications

References 50 publications

Admm Slim

Admm Slim

Explainable Recommendation: A Survey and New Perspectives

Evaluation Infrastructures for Academic Shared Tasks

Contact Info

Product

Resources

About