Studying retrievability of publications and datasets in an integrated retrieval system

Abstract

In this paper, we investigate the retrievability of datasets and publications in a real-life Digital Library (DL). The measure of retrievability was originally developed to quantify the influence that a retrieval system has on the access to information. Retrievability can also enable DL engineers to evaluate their search engine to determine the ease with which the content in the collection can be accessed. Following this methodology, in our study, we propose a system-oriented approach for studying dataset and publication retrieval. A speciality of this paper is the focus on measuring the accessibility biases of various types of DL items and including a metric of usefulness. Among other metrics, we use Lorenz curves and Gini coefficients to visualize the differences of the two retrievable document types (specifically datasets and publications). Empirical results reported in the paper show a distinguishable diversity in the retrievability scores among the documents of different types.

Retrievability in an integrated retrieval system: an extended study

“…This factor additionally introduces popularity bias into the reproducibility of documents in the form of query popularity. The results and observations reported in our earlier study (Roy et al, 2022) were based on this type of interaction log. In order to exclusively understand the reproducibility without the query popularity factor, we have only considered unique queries in this work.…”

“…We verify the research questions put forward and discussed by Roy et al (2022) in the updated system with a variety of item types tested with more queries (see Section 4). Similar to the previous work, we substantiate the following research questions on the integrated search system GESIS Search focusing on an additional type of item: Variables together with Publication and Dataset:…”

“…Our previous study (Roy et al, 2022) was designed to take all queries in the query log into account. This had the benefit of being as close to the real search behaviour as possible.…”

“…This ensures realistic queries in Q of Equation 1 as opposed to the simulated queries used in Azzopardi and Vinay (2008) or Traub et al (2016). The data used in this study is an extended version of our previous work (Roy et al, 2022); in this log, all the interactions of real users with the search system were recorded for a period of more than five years, specifically between July 2017 and July 2022. The log records more than 2.3 million queries submitted to the integrated search system.…”

“…As an extension of our prior work (Roy et al, 2022), we investigate the retrievability of various types of documents in an integrated digital library GESIS Search (see Section 3), focusing on various types of data, particularly datasets, publications and variables. The assumption followed here is that in an ideal ranking system 1 , the retrievability of each indexed item (dataset or other publication) is equally distributed.…”