A comparative analysis of offline and online evaluations and discussion of research paper recommender system evaluation

Beel, Joeran; Genzmehr, Marcel; Langer, Stefan; Nürnberger, Andreas; Gipp, Béla

doi:10.1145/2532508.2532511

Cited by 106 publications

(82 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The majority (71 %) of approaches were evaluated using offline evaluations, which are subject to various shortcomings. Some claim that offline evaluations should not be used for evaluating research-paper recommender systems [191]. If that is true, most of the reviewed evaluations would be of little significance.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Research-paper recommender systems: a literature survey

et al. 2015

Self Cite

View full text Add to dashboard Cite

In the last 16 years, more than 200 research articles were published about research-paper recommender systems. We reviewed these articles and present some descriptive statistics in this paper, as well as a discussion about the major advancements and shortcomings and an overview of the most common recommendation concepts and approaches. We found that more than half of the recommendation approaches applied content-based filtering (55 %). Collaborative filtering was applied by only 18 % of the reviewed approaches, and graph-based recommendations by 16 %. Other recommendation concepts included stereotyping, item-centric recommendations, and hybrid recommendations. The content-based filtering approaches mainly utilized papers that the users had authored, tagged, browsed, or downloaded. TF-IDF was the most frequently applied weighting scheme. In addition to simple terms, n-grams, topics, and citations were utilized to model users' information needs. Our review revealed some shortcomings of the current research. First, it remains unclear which recommendation concepts and approaches are the most promising. For instance, researchers reported different results on the performance of contentbased and collaborative filtering. Sometimes content-based filtering performed better than collaborative filtering and Linnaeus University, Kalmar, Sweden sometimes it performed worse. We identified three potential reasons for the ambiguity of the results. (A) Several evaluations had limitations. They were based on strongly pruned datasets, few participants in user studies, or did not use appropriate baselines. (B) Some authors provided little information about their algorithms, which makes it difficult to re-implement the approaches. Consequently, researchers use different implementations of the same recommendations approaches, which might lead to variations in the results. (C) We speculated that minor variations in datasets, algorithms, or user populations inevitably lead to strong variations in the performance of the approaches. Hence, finding the most promising approaches is a challenge. As a second limitation, we noted that many authors neglected to take into account factors other than accuracy, for example overall user satisfaction. In addition, most approaches (81 %) neglected the user-modeling process and did not infer information automatically but let users provide keywords, text snippets, or a single paper as input. Information on runtime was provided for 10 % of the approaches. Finally, few research papers had an impact on research-paper recommender systems in practice. We also identified a lack of authority and long-term research interest in the field: 73 % of the authors published no more than one paper on research-paper recommender systems, and there was little cooperation among different co-author groups. We concluded that several actions could improve the research landscape: developing a common evaluation framework, agreement on the information to include in research papers, a stronger focus on non-accuracy aspect...

show abstract

Section: Discussionmentioning

confidence: 99%

“…3). [ [5][6][7][8]20,25,69,70,112,113,125,159,[189][190][191][192][193][194] We then present the survey of the 96 approaches' evaluations (Sect. 4), followed by an analysis of the 62 approaches that we chose for the in-depth review (Sect.…”

Section: Paper | Article | Citation] [Recommender | Recommendation] [mentioning

confidence: 99%

Research-paper recommender systems: a literature survey

et al. 2015

Self Cite

View full text Add to dashboard Cite

show abstract

“…Typically, offline evaluations are considered suitable to preselect a set of promising algorithms, which are subsequently evaluated in online evaluations or by a user study [11]. However, there is serious criticism of offline evaluations [60][61][62][63][64][65]106,111].…”

Section: Evaluating Methodsmentioning

confidence: 99%

Research paper recommender system evaluation

Beel¹,

Langer²,

Genzmehr³

et al. 2013

Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation

Self Cite

102

View full text Add to dashboard Cite

Over 80 approaches for academic literature recommendation exist today. The approaches were introduced and evaluated in more than 170 research articles, as well as patents, presentations and blogs. We reviewed these approaches and found most evaluations to contain major shortcomings. Of the approaches proposed, 21% were not evaluated. Among the evaluated approaches, 19% were not evaluated against a baseline. Of the user studies performed, 60% had 15 or fewer participants or did not report on the number of participants. Information on runtime and coverage was rarely provided. Due to these and several other shortcomings described in this paper, we conclude that it is currently not possible to determine which recommendation approaches for academic literature are the most promising. However, there is little value in the existence of more than 80 approaches if the best performing approaches are unknown.

show abstract

“…Filtering these information sources result in the higher accuracy of recommendations. In [8], authors have discussed the online and offline evaluation of research paper recommender framework and conclude that offline evaluation in this domain does not provide promising results. Docear"s research paper recommender framework is proposed using content-based filtering in which user"s data (citations, references, and papers) is directed in mind maps and are then utilized for recommendations [9].…”

Section: Literature Reviewmentioning

confidence: 99%

Recommender System for Journal Articles using Opinion Mining and Semantics

Sardar¹,

Ferzund²,

Suryani³

et al. 2017

ijacsa

View full text Add to dashboard Cite

Abstract-Till date, the dominant part of Recommender Systems (RS) work focusing on single domain, i.e. for films, books and shopping and so on. However, human inclinations may traverse over numerous areas. Thus, utilization practices on related things from various domains can be valuable for RS to make recommendations. Academic articles, such as research papers are the way to express ideas and thoughts for the research community. However, there have been a lot of journals available which recognize these technical writings. In addition, journal selection procedure should consider user experience about the journals in order to recommend users most relevant journal. In this work of journal recommendation system, the data about the user experience targeting various aspects of journals has been gathered which addresses user experience about any journal. In addition, data set of archive articles has been developed considering the user experience in this regard. Moreover, the user experience and gathered data of archives are analyzed using two different frameworks based on semantics in order to have better consolidated recommendations. Before submission, we offer services on behalf of the research community that exploit user reviews and relevant data to suggest suitable journal according to the needs of the author.

show abstract

A comparative analysis of offline and online evaluations and discussion of research paper recommender system evaluation

Cited by 106 publications

References 25 publications

Research-paper recommender systems: a literature survey

Research-paper recommender systems: a literature survey

Research paper recommender system evaluation

Recommender System for Journal Articles using Opinion Mining and Semantics

Contact Info

Product

Resources

About