An Adaptive Systems Approach to the Implementation and Evaluation of Digital Library Recommendation Systems

Bollen, Johan; Rocha, Luís M.

doi:10.1007/3-540-45268-0_39

Cited by 6 publications

(2 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Herlocker et al (2004) wrote an article on how to evaluate collaborative-filtering approaches. Various authors showed that offline and online evaluations often provide contradictory results (Cremonesi et al 2012;McNee et al 2002), and several more papers about various aspects of recommender-system evaluation have been published (Amatriain et al 2009;Beel and Langer 2015;Bollen and Rocha 2000;Bogers and van den Bosch 2007;Cremonesi et al 2011;Domingues Garcia et al 2012;Ge et al 2010;Hayes et al 2002;Hofmann et al 2014;Jannach et al 2012;Knijnenburg et al 2011Knijnenburg et al , 2012Konstan and Riedl 2012;Manouselis and Verbert 2013;Pu et al 2011Pu et al , 2012Said 2013;Shani and Gunawardana 2011). However, while many of the findings in these papers are important with respect to reproducibility, the authors did not mention or discuss their findings in the context of reproducibility.…”

Section: Introductionmentioning

confidence: 99%

Towards reproducibility in recommender-systems research

Beel

Breitinger

Langer

et al. 2016

User Model User-Adap Inter

View full text Add to dashboard Cite

Numerous recommendation approaches are in use today. However, comparing their effectiveness is a challenging task because evaluation results are rarely reproducible. In this article, we examine the challenge of reproducibility in recommender-system research. We conduct experiments using Plista's news recommender system, and Docear's research-paper recommender system. The experiments show that there are large discrepancies in the effectiveness of identical recommendation approaches in only slightly different scenarios, as well as large discrepancies for slightly different approaches in identical scenarios. For example, in one newsrecommendation scenario, the performance of a content-based filtering approach User-Adapted Interaction : umuai ; 26 (2016), 1. -S. 69-101 https://dx.doi.org/10.1007 was twice as high as the second-best approach, while in another scenario the same content-based filtering approach was the worst performing approach. We found several determinants that may contribute to the large discrepancies observed in recommendation effectiveness. Determinants we examined include user characteristics (gender and age), datasets, weighting schemes, the time at which recommendations were shown, and user-model size. Some of the determinants have interdependencies. For instance, the optimal size of an algorithms' user model depended on users' age. Since minor variations in approaches and scenarios can lead to significant changes in a recommendation approach's performance, ensuring reproducibility of experimental results is difficult. We discuss these findings and conclude that to ensure reproducibility, the recommender-system community needs to (1) survey other research fields and learn from them, (2) find a common understanding of reproducibility, (3) identify and understand the determinants that affect reproducibility, (4) conduct more comprehensive experiments, (5) modernize publication practices, (6) foster the development and use of recommendation frameworks, and (7) establish best-practice guidelines for recommender-systems research.

show abstract

Section: Introductionmentioning

confidence: 99%

Towards reproducibility in recommender-systems research

Beel

Breitinger

Langer

et al. 2016

User Model User-Adap Inter

View full text Add to dashboard Cite

show abstract

“…The reverse is true for IV 1 . Thus, the evidence set obtained from (10) contains IV c with the lowest evidence, and IV 1 with the highest.…”

Section: The Linguistic "And/or" Combinationmentioning

confidence: 94%

Combination of evidence in recommendation systems characterized by distance functions

Rocha

2002 IEEE World Congress on Computational Intelligence. 2002 IEEE International Conference on Fuzzy Systems. FUZZ-IEEE'02. Proc

View full text Add to dashboard Cite

-Recommendation systems for different Document Networks (DN) such as the World Wide Web (WWW), DigitalLibarries, or Scientific Databases, often make use of distance functions extracted from relationships among documents and between documents and semantic tags. For instance, documents in the WWW are related via a hyperlink network, while documents in bibliographic databases are related by citation and collaboration networks. Furthermore, documents can be related to semantic tags such as keywords used to describe their content. The distance functions computed from these relations establish associative networks among items of the DN, and allow recommendation systems to identify relevant associations for individual users. The process of recommendation can be improved by integrating associative data from different sources. Thus we are presented with a problem of combining evidence (about associations between items) from different sources characterized by distance functions. In this paper we summarize our work on (1) inferring associations from semi-metric distance functions and (2) combining evidence from different (distance) associative DN. RECOMMENDATION IN DOCUMENT NETWORKSThe prime example of a Document Network (DN) is the World Wide Web (WWW). But many other types of such networks exist: bibliographic databases containing scientific publications, preprints, internal reports, as well as databases of datasets used in scientific endeavors. Each of these databases possesses several distinct relationships among documents and between documents and semantic tags or indices that classify documents appropriately.DN typically function as information resources for communities of users who query them to obtain relevant information for their activities. Resources such as the Internet, Digital Libraries, and the like have become ubiquitous in the past decade, demanding the development of new techniques to cater to the information needs of communities of users. These techniques come from the field of Information Retrieval, and are typically known as Recommender Systems e.g. The algorithms we have developed in this area integrate evidence about the association amongst elements of DN, amongst users, and about the interests of individual users and their communities. In particular, a soft computing algorithm (TalkMine) has been created to integrate such evidence and also adapt DN to the expectations of their users [15]. The process of integration of knowledge in TalkMine requires the construction of distance functions on DN that characterize the associations amongst their components. Below we discuss how such distance functions are used to characterize DN and for recommendation. DISTANCE FUNCTIONS IN DOCUMENT NETWORKS Harvesting Relations from Document NetworksFor each DN we can identify several distinct relations among documents and between documents and semantic tags used to classify documents appropriately. For instance, documents in the WWW are related via a hyperlink network, while documents in bibliographic database...

show abstract