Overview of LiLAS 2021 – Living Labs for Academic Search

Schaer, Philipp; Breuer, Timo; Castro, Leyla Jael; Wolff, Benjamin; Schaible, Johann; Tavakolpoursaleh, Narges

doi:10.1007/978-3-030-85251-1_25

Cited by 5 publications

(14 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Compared to conventional Cranfield paradigm information retrieval (IR) experiments, Living Lab IR experiments aim to evaluate search systems in a real-world fashion through actual user interactions. Therefore, experimental systems extend existing search systems and are evaluated based on collected user feedback [20]. By that, the Living Labs for Academic Search (LiLAS) lab followed a series of labs dedicated to the living lab approach like NewsREEL [16], LL4IR [23] and TREC OpenSearch [4] did before.…”

Section: Related Workmentioning

confidence: 99%

“…As most useful appeared to use the TF-IDF scores from the live system directly as relevance scores in the pseudo test collection. This pseudo test collection allowed to compare our experimental system with the TF-IDF based live system as baseline offline [20].…”

Section: System Pre-testingmentioning

confidence: 99%

“…Schaer et al provide a comprehensive evaluation of the different systems, including also weighted results accounting for the interleaved experiment setup where two systems merge their results into one result page ranking [20]. Additionally, this analysis focuses more specifically on the recommender system itself.…”

Section: Experimental Evaluationmentioning

confidence: 99%

“…The proposed system makes use of the broad similarities between scientific publications and research datasets and is based on the probabilistic BM25 ranking function to determine the similarity between index and query [18]. Results from the TREC-COVID Challenge 1 described by Roberts et al show that almost The system was evaluated by the Living Labs for Academic Search (LiLAS) CLEF Challenge as a type B submission to Task 2 [20]. While task one focused on ad-hoc retrieval of scientific publications, Task 2 demanded for dataset recommendations related to scientific publications.…”

Section: Introductionmentioning

confidence: 99%

“…The layout of the website is shown in Figure 1. More detailed task explanations and the general evaluation can be found in the lab overview [20]. Living Labs differ strongly from retrieval experiments following the Cranfield paradigm and bring several unique challenges.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Evaluating Research Dataset Recommendations in a Living Lab

Keller

Munz

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The search for research datasets is as important as laborious. Due to the importance of the choice of research data in further research, this decision must be made carefully. Additionally, because of the growing amounts of data in almost all areas, research data is already a central artifact in empirical sciences. Consequentially, research dataset recommendations can beneficially supplement scientific publication searches. We formulated the recommendation task as a retrieval problem by focussing on broad similarities between research datasets and scientific publications. In a multistage approach, initial recommendations were retrieved by the BM25 ranking function and dynamic queries. Subsequently, the initial ranking was re-ranked utilizing click feedback and document embeddings. The proposed system was evaluated live on real user interaction data using the STELLA infrastructure in the LiLAS Lab at CLEF 2021. Our experimental system could efficiently be fine-tuned before the live evaluation by pre-testing the system with a pseudo test collection based on prior user interaction data from the live system. The results indicate that the experimental system outperforms the other participating systems.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: System Pre-testingmentioning

confidence: 99%

Section: Experimental Evaluationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Evaluating Research Dataset Recommendations in a Living Lab

Keller

Munz

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

What Happened in CLEF$$\ldots $$ For Another While?

Ferro

2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Scientific paper recommendation systems: a literature review of recent publications

Kreutz

Schenkel

2022

Int J Digit Libr

View full text Add to dashboard Cite

Scientific writing builds upon already published papers. Manual identification of publications to read, cite or consider as related papers relies on a researcher’s ability to identify fitting keywords or initial papers from which a literature search can be started. The rapidly increasing amount of papers has called for automatic measures to find the desired relevant publications, so-called paper recommendation systems. As the number of publications increases so does the amount of paper recommendation systems. Former literature reviews focused on discussing the general landscape of approaches throughout the years and highlight the main directions. We refrain from this perspective, instead we only consider a comparatively small time frame but analyse it fully. In this literature review we discuss used methods, datasets, evaluations and open challenges encountered in all works first released between January 2019 and October 2021. The goal of this survey is to provide a comprehensive and complete overview of current paper recommendation systems.

show abstract

Overview of LiLAS 2021 – Living Labs for Academic Search

Cited by 5 publications

References 11 publications

Evaluating Research Dataset Recommendations in a Living Lab

Evaluating Research Dataset Recommendations in a Living Lab

What Happened in CLEF$$\ldots $$ For Another While?

Scientific paper recommendation systems: a literature review of recent publications

Contact Info

Product

Resources

About