Online Evaluations for Everyone: Mr. DLib’s Living Lab for Scholarly Recommendations

Abstract

We introduce the first 'living lab' for scholarly recommender systems. This lab allows recommender-system researchers to conduct online evaluations of their novel algorithms for scholarly recommendations, i.e., recommendations for research papers, citations, conferences, research grants, etc. Recommendations are delivered through the living lab's API to platforms such as reference management software and digital libraries. The living lab is built on top of the recommender-system as-a-service Mr. DLib. Current partners are the reference management software JabRef and the CORE research team. We present the architecture of Mr. DLib's living lab as well as usage statistics on the first sixteen months of operating it. During this time, 1,826,643 recommendations were delivered with an average click-through rate of 0.21%.

Scientific paper recommendation systems: a literature review of recent publications

“…systems. This could for example be realised in an extendable paper recommendation benchmarking system similar to the in a living lab environments ArXivDigest [36], Mr. DLib's living lab [14] or LiLAS [91] but with the additional property that it also provides build-in offline evaluations. As fairness and explainability of current paper recommendation systems have not been tackled widely, those aspects should be further explored.…”

“…Living labs help researchers conduct meaningful evaluations by providing an environment, in which recommendations produced by experimental systems are shown to real users in realistic scenarios [14]. We found three relevant works for the area of scientific paper recommendation: Beel et al [14] proposed a living lab for scholarly recommendation built on top of Mr. DLib, their recommender-as-a-service system. They log users' actions such as clicks, downloads and purchases for related recommended papers.…”

“…User studies where participants rate recommendations according to some criteria and are aware of the study are lab-based, all others are considered realworld studies. Living labs [14,36,91] for example enable realworld user studies. On average the lab-based user studies were conducted with 17.83 users.…”

“…Noteworthy counter-examples 31 are the studies by Bulut et al [22] who emailed 50 researchers to rate relevancy of recommended articles or Chaudhuri et al [26] who asked 45 participants to rate their acceptance of recommended publications. Another option to overcome this issue is utilisation of living labs as seen with ArXivDigest [36], Mr. DLib's living lab [14] or LiLAS for the related tasks of dataset recommendation for scientific publications and multi-lingual document retrieval [91].…”

RARD: The Related-Article Recommendation Dataset

“…To generate a set of recommendation, Mr. DLib harnesses different recommendation approaches including content-based filtering. Mr. DLib also uses external recommendation APIs such as the CORE Recommendation API [29,30] as part of a 'living lab' [31]. The algorithm selection and parametrization is managed by Mr. DLib's A/B testing engine.…”