A framework for designing retrieval effectiveness studies of library information systems using human relevance assessments

Abstract

Purpose The purpose of this paper is to demonstrate how to apply traditional information retrieval (IR) evaluation methods based on standards from the Text REtrieval Conference and web search evaluation to all types of modern library information systems (LISs) including online public access catalogues, discovery systems, and digital libraries that provide web search features to gather information from heterogeneous sources. Design/methodology/approach The authors apply conventional procedures from IR evaluation to the LIS context considering the specific characteristics of modern library materials. Findings The authors introduce a framework consisting of five parts: search queries, search results, assessors, testing, and data analysis. The authors show how to deal with comparability problems resulting from diverse document types, e.g., electronic articles vs printed monographs and what issues need to be considered for retrieval tests in the library context. Practical implications The framework can be used as a guideline for conducting retrieval effectiveness studies in the library context. Originality/value Although a considerable amount of research has been done on IR evaluation, and standards for conducting retrieval effectiveness studies do exist, to the authors’ knowledge this is the first attempt to provide a systematic framework for evaluating the retrieval effectiveness of twenty-first-century LISs. The authors demonstrate which issues must be considered and what decisions must be made by researchers prior to a retrieval test.

Using the Kano Model and the Quality of the Publishing Function in Evaluating Jordan's Academic Digital Libraries: A Case Study of the University of Jordan

“…One of the studies reviewed for designing this questionnaire is the Mexican Library Association's study that focused on assessing information retrieval from the point of view of users and the extent to which they meet their needs (Behnert & Lewandowski, 2017). Then Beck study (2005), which focused on some important features and characteristics of digital libraries such as: reliability of retrieved information, citation, integrity and objectivity, site maintenance, and the extent of coverage of available information.…”

Financial literacy resources in US public libraries: website analysis

“…One common alternative to individual assessments of relevancy is to rely on the "wisdom of crowds" either through user studies or via click-through rates, which appears to improve the stability of relevancy judgments (Zhitomirsky-Geffet et al, 2016;Zhitomirsky-Geffet et al, 2018). Studies evaluating the information retrieval of library discovery tools have often relied on user observation in laboratory settings, log analysis or comparison of search results to student citations in particular projects or courses (Behnert and Lewandowski, 2017;Galbreath et al, 2021). None of these options were feasible for this study.…”

Usability Evaluation for Online Professional Search in the Dutch Archaeology Domain

“…In usability studies for IR systems, the most used evaluation protocol is to provide the users with a number of information problems and ask them to solve these problems using the search system at hand. A questionnaire is used after the process to assess their satisfaction (Spink, 2002;Behnert and Lewandowski, 2017;Rico et al, 2019).…”