Learning-to-Rank and Relevance Feedback for Literature Appraisal in Empirical Medicine

Lagopoulos, Athanasios; Anagnostou, Antonios; Minas, Adamantios; Tsoumakas, Grigorios

doi:10.1007/978-3-319-98932-7_5

Cited by 14 publications

(6 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There has been significant interest in the development of techniques to support the identification of studies for inclusion in systematic reviews by applying text mining techniques, for example, [9][10][11][12][13][14][15][16]. The vast majority of this work has focused on the identification of studies for new reviews and only a few papers explored the problem of identifying relevant studies for review updates [9,17,18].…”

Section: Related Workmentioning

confidence: 99%

Refining Boolean queries to identify relevant studies for systematic review updates

Alharbi

Stevenson

2020

Journal of the American Medical Informatics Association

View full text Add to dashboard Cite

Objective Systematic reviews are important in health care but are expensive to produce and maintain. The authors explore the use of automated transformations of Boolean queries to improve the identification of relevant studies for updates to systematic reviews. Materials and Methods A set of query transformations, including operator substitution, query expansion, and query reduction, were used to iteratively modify the Boolean query used for the original systematic review. The most effective transformation at each stage is identified using information about the studies included and excluded from the original review. A dataset consisting of 22 systematic reviews was used for evaluation. Updated queries were evaluated using the included and excluded studies from the updated version of the review. Recall and precision were used as evaluation measures. Results The updated queries were more effective than the ones used for the original review, in terms of both precision and recall. The overall number of documents retrieved was reduced by more than half, while the number of relevant documents found increased by 10.3%. Conclusions Identification of relevant studies for updates to systematic reviews can be carried out more effectively by using information about the included and excluded studies from the original review to produce improved Boolean queries. These updated queries reduce the overall number of documents retrieved while also increasing the number of relevant documents identified, thereby representing a considerable reduction in effort required by systematic reviewers.

show abstract

Section: Related Workmentioning

confidence: 99%

Refining Boolean queries to identify relevant studies for systematic review updates

Alharbi

Stevenson

2020

Journal of the American Medical Informatics Association

View full text Add to dashboard Cite

show abstract

“…For the implementation of the AUTO-TAR BMI, we followed Algorithm 1 as described in Cormack and Grossman (2018). The Hybrid variation method is similar to the one presented in Lagopoulos et al (2018) which uses TF-IDF instead of the Sent2Vec for the representation of the documents in the intra-review model. Finally, for the PubMed method, we query the PubMed database 9 , using the Entrez Programming Utilities, with the title and the objective of each SLR.…”

Section: Comparison With State-of-the-art Methods and A Baselinementioning

confidence: 99%

“…This work is an extension of a previously published conference paper (Lagopoulos et al, 2018). Specifically, we extended our previous pipeline with a primary retrieval engine, fine-tuned the inter-review ranker features and adopted sentence embeddings for both inter-review and intra-review rankers.…”

Section: Introductionmentioning

confidence: 99%

From Protocol to Screening: A Hybrid Learning Approach for Technology-Assisted Systematic Literature Reviews

Lagopoulos,

Tsoumakas

2020

Preprint

Self Cite

View full text Add to dashboard Cite

In the medical domain, a Systematic Literature Review (SLR) attempts to collect all empirical evidence, that fit pre-specified eligibility criteria, in order to answer a specific research question. The process of preparing an SLR consists of multiple tasks that are labor-intensive and time-consuming, involving large monetary costs. Technology-assisted review (TAR) methods automate the different processes of creating an SLR and they are particularly focused on reducing the burden of screening for reviewers. We present a novel method for TAR that implements a full pipeline from the research protocol to the screening of the relevant papers. Our pipeline overcomes the need of a Boolean query constructed by specialists and consists of three different components: the primary retrieval engine, the inter-review ranker and the intra-review ranker, combining learning-to-rank techniques with a relevance feedback method. In addition, we contribute an updated version of the Task 2 of the CLEF 2019 eHealth Lab dataset, which we make publicly available. Empirical results on this dataset show that our approach can achieve state-ofthe-art results.

show abstract

“…This ranking of studies has come to be known as 'screening prioritisation', as popularised by the CLEF TAR tasks which aimed to automate these early stages of the systematic review creation pipeline [9,11,10]. As a result, in recent years there has been an uptake in Information Retrieval approaches to enable screening prioritisation [18,5,3,25,2,22,16,15,1,27,21]. The vast majority of screening prioritisation use a different representation than the original Boolean query for ranking.…”

Section: Related Workmentioning

confidence: 99%

Seed-driven Document Ranking for Systematic Reviews: A Reproducibility Study

Wang¹,

Scells²,

Mourad³

et al. 2021

Preprint

View full text Add to dashboard Cite

Screening or assessing studies is critical to the quality and outcomes of a systematic review. Typically, a Boolean query retrieves the set of studies to screen. As the set of studies retrieved is unordered, screening all retrieved studies is usually required for high-quality systematic reviews. Screening prioritisation, or in other words, ranking the set of studies, enables downstream activities of a systematic review to begin in parallel. We investigate a method that exploits seed studies -potentially relevant studies used to seed the query formulation process -for screening prioritisation. Our investigation aims to reproduce this method to determine if it is generalisable on recently published datasets and determine the impact of using multiple seed studies on effectiveness. We show that while we could reproduce the original methods, we could not replicate their results exactly. However, we believe this is due to minor differences in document pre-processing, not deficiencies with the original methodology. Our results also indicate that our reproduced screening prioritisation method, (1) is generalisable across datasets of similar and different topicality compared to the original implementation, (2) that when using multiple seed studies, the effectiveness of the method increases using our techniques to enable this, (3) and that the use of multiple seed studies produces more stable rankings compared to single seed studies. Finally, we make our implementation and results publicly available at the following URL: https://github.com/ielab/sdr.

show abstract

Learning-to-Rank and Relevance Feedback for Literature Appraisal in Empirical Medicine

Cited by 14 publications

References 17 publications

Refining Boolean queries to identify relevant studies for systematic review updates

Refining Boolean queries to identify relevant studies for systematic review updates

From Protocol to Screening: A Hybrid Learning Approach for Technology-Assisted Systematic Literature Reviews

Seed-driven Document Ranking for Systematic Reviews: A Reproducibility Study

Contact Info

Product

Resources

About