OpenSearch

Jagerman, Rolf; Balog, Krisztian; Rijke, Maarten de

doi:10.1145/3239575

Cited by 16 publications

(10 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, the first living lab for ad-hoc retrieval was held at CLEF in 2015 and was continued in a second iteration in 2016 [83]. The same organizers were also involved in the Open Search track at TREC in 2016 and 2017 [42]. NEWSREEL was the first living lab for real-time news recommendations and ran from 2014 until 2017 [15,41].…”

Section: Living Labsmentioning

confidence: 99%

“…The infrastructure was tailored explicitly for shared task collaborations and was the backbone of the LiLAS lab at CLEF in 2021. One of the substantial improvements over earlier living lab attempts is the possibility of submitting the entire experimental system instead of submitting pre-computed results only, which addresses the shortcoming of pre-computed results in earlier living labs [42].…”

Section: Living Labsmentioning

confidence: 99%

“…In order to pave the way for experimental evaluations in small-to mid-scale user data environments, the living lab paradigm [42] was introduced. As part of these efforts, small and domainspecific search services with different applications like product or academic search opened their infrastructures for researchers who are able to evaluate their IR systems in online experiments with user interaction data.…”

Section: Introductionmentioning

confidence: 99%

“…Earlier works concluded that user interaction data in living labs is sparse [42,81,83]. While there is a need to validate laboratory or system-oriented experiments in the real world, the corresponding experiments come with the risk of harming the user experience.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Validating Synthetic Usage Data in Living Lab Environments

Breuer,

Fuhr,

Schaer

2024

J. Data and Information Quality

View full text Add to dashboard Cite

Evaluating retrieval performance without editorial relevance judgments is challenging, but instead, user interactions can be used as relevance signals. Living labs offer a way for small-scale platforms to validate information retrieval systems with real users. If enough user interaction data is available, click models can be parameterized from historical sessions to evaluate systems before exposing users to experimental rankings. However, interaction data is sparse in living labs, and little is studied about how click models can be validated for reliable user simulations when click data is available in moderate amounts. This work introduces an evaluation approach for validating synthetic usage data generated by click models in data-sparse human-in-the-loop environments like living labs. We ground our methodology on the click model’s estimates about a system ranking compared to a reference ranking for which the relative performance is known. Our experiments compare different click models and their reliability and robustness as more session log data becomes available. In our setup, simple click models can reliably determine the relative system performance with already 20 logged sessions for 50 queries. In contrast, more complex click models require more session data for reliable estimates, but they are a better choice in simulated interleaving experiments when enough session data is available. While it is easier for click models to distinguish between more diverse systems, it is harder to reproduce the system ranking based on the same retrieval algorithm with different interpolation weights. Our setup is entirely open, and we share the code to reproduce the experiments.

show abstract

Section: Living Labsmentioning

confidence: 99%

Section: Living Labsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Validating Synthetic Usage Data in Living Lab Environments

Breuer,

Fuhr,

Schaer

2024

J. Data and Information Quality

View full text Add to dashboard Cite

show abstract

“…LiLAS offers two different evaluation tasks: Academic ad-hoc retrieval for the multi-lingual and multi-source Life Science search portal LIVIVO and research data recommendation within the Social Science portal GESIS Search. For both tasks, participants are invited to submit -Type A pre-computed runs based on previously compiled queries (ad-hoc search) or documents (research data recommendations) from server logs (comparable to the CLEF LL4IR or TREC Open Search labs [8]) or -Type B Docker containers of full running retrieval/recommendation systems that run within our evaluation framework called STELLA.…”

Section: Evaluation Infrastructure and Submission Typesmentioning

confidence: 99%

Living Lab Evaluation for Life and Social Sciences Search Platforms - LiLAS at CLEF 2021

Schaer

Schaible

Castro

2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Meta-evaluation studies of system performances in controlled offline evaluation campaigns, like TREC and CLEF, show a need for innovation in evaluating IR-systems. The field of academic search is no exception to this. This might be related to the fact that relevance in academic search is multilayered and therefore the aspect of user-centric evaluation is becoming more and more important. The Living Labs for Academic Search (LiLAS) lab aims to strengthen the concept of usercentric living labs for the domain of academic search by allowing participants to evaluate their retrieval approaches in two real-world academic search systems from the life sciences and the social sciences. To this end, we provide participants with metadata on the systems' content as well as candidate lists with the task to rank the most relevant candidate to the top. Using the STELLA-infrastructure, we allow participants to easily integrate their approaches into the real-world systems and provide the possibility to compare different approaches at the same time.

show abstract