Offline evaluations are the most common evaluation method for research paper recommender systems. However, no thorough discussion on the appropriateness of offline evaluations has taken place, despite some voiced criticism. We conducted a study in which we evaluated various recommendation approaches with both offline and online evaluations. We found that results of offline and online evaluations often contradict each other. We discuss this finding in detail and conclude that offline evaluations may be inappropriate for evaluating research paper recommender systems, in many settings.
In this demo paper we present Docear's research paper recommender system. Docear is an academic literature suite to search, organize, and create research articles. The users' data (papers, references, annotations, etc.) is managed in mind maps and these mind maps are utilized for the recommendations. Using content-based filtering methods, Docear's recommender achieves click-through rates around 6%, in some scenarios even over 10%.
In this paper we show that organic recommendations are preferred over commercial recommendations even when they point to the same freely downloadable research papers. Simply the fact that users perceive recommendations as commercial decreased their willingness to accept them. It is further shown that the exact labeling of recommendations matters. For instance, recommendations labeled as 'advertisement' performed worse than those labeled as 'sponsored'. Similarly, recommendations labeled as 'Free Research Papers' performed better than those labeled as 'Research Papers'. However, whatever the differences between the labels were -the best performing recommendations were those with no label at all.
Abstract. Mind-maps have been widely neglected by the information retrieval (IR) community. However, there are an estimated two million active mind-map users, who create 5 million mind-maps every year, of which a total of 300,000 is publicly available. We believe this to be a rich source for information retrieval applications, and present eight ideas on how mind-maps could be utilized by them. For instance, mind-maps could be utilized to generate user models for recommender systems or expert search, or to calculate relatedness of web-pages that are linked in mind-maps. We evaluated the feasibility of the eight ideas, based on estimates of the number of available mind-maps, an analysis of the content of mind-maps, and an evaluation of the users' acceptance of the ideas. We concluded that user modelling is the most promising application with respect to mind-maps. A user modelling prototype -a recommender system for the users of our mind-mapping software Docear -was implemented, and evaluated. Depending on the applied user modelling approaches, the effectiveness, i.e. click-through rate on recommendations, varied between 0.28% and 6.24%. This indicates that mind-map based user modelling is promising, but not trivial, and that further research is required to increase effectiveness.Keywords: mind-maps, content analysis, user modelling, information retrieval IntroductionInformation retrieval (IR) applications utilize many items beyond the items' original purpose. For instance, emails are intended as a means of communication, but Google utilizes them for generating user profiles and displaying personalized advertisement [1]; social tags can help to organize private web-page collections, but search engines utilize them for indexing websites [2]; research articles are meant to publish research results, but they, or more precisely their references, are utilized to analyze the impact of researchers and institutions [3]. We propose that mind-maps are an equally valuable source for information retrieval as are social tags, emails, research articles, etc. Consequently, our research objective was to identify, how mind-maps could be used to empower IR applications. To Preprint, to be published at UMAP 2014. Downloaded from http://docear.org
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.