Evaluation Infrastructures for Academic Shared Tasks

Abstract

Academic search systems aid users in finding information covering specific topics of scientific interest and have evolved from early catalog-based library systems to modern web-scale systems. However, evaluating the performance of the underlying retrieval approaches remains a challenge. An increasing amount of requirements for producing accurate retrieval results have to be considered, e.g., close integration of the system's users. Due to these requirements, small to mid-size academic search systems cannot evaluate their retrieval system in-house. Evaluation infrastructures for shared tasks alleviate this situation. They allow researchers to experiment with retrieval approaches in specific search and recommendation scenarios without building their own infrastructure. In this paper, we elaborate on the benefits and shortcomings of four state-of-the-art evaluation infrastructures on search and recommendation tasks concerning the following requirements: support for online and offline evaluations, domain specificity of shared tasks, and reproducibility of experiments and results. In addition, we introduce an evaluation infrastructure concept design aiming at reducing the shortcomings in shared tasks for search and recommender systems.

Overview of LiLAS 2021 -- Living Labs for Academic Search

“…Following previous living lab experiments, we implement the interleaving method by the Team-Draft-Interleaving algorithm [12]. More specifically, we refactored exactly the same implementation 13 for the highest degree of comparability. Furthermore we follow Gingstad et al's proposal of a weighted score based on click events [5] and define the Reward as…”

“…Progress in the field of academic search and its corresponding domains is usually evaluated by means of shared tasks that are based on the principles of Cranfield/TREC-style studies [13]. Most recently the TREC-COVID evaluation campaign run by NIST attracted a high number of participants and showed the high impact of scientific retrieval tasks in the community.…”

Evaluating Research Dataset Recommendations in a Living Lab

“…Through the STELLA infrastructure the experimental systems of both types could be integrated into the live systems. Further, STELLA creates an interleaved ranking by systematically combining the results from two systems [8,22]. More lifelike results and insights are expected by utilizing real user interactions to assess search systems.…”

Overview of LiLAS 2020 – Living Labs for Academic Search

“…The Living Labs for Academic Search (LiLAS) workshop fosters the discussion, research, and evaluation of academic search systems, and it employs the concept of living labs to the domain of academic search [13]. The goal is to expand the knowledge on improving the search for academic resources like literature, research data, and the interlinking between these resources.…”