Seed-Driven Document Ranking for Systematic Reviews: A Reproducibility Study

Wang, Shuai; Scells, Harrisen; Mourad, Ahmed; Zuccon, Guido

doi:10.1007/978-3-030-99736-6_46

Cited by 9 publications

(13 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Bag of clinical words where clinical terms in a study are used. Bag of words representations are more effective than the bag of clinical words representation [34]. Therefore, we adopt the bag of words representation (as indicated by BOW) here.…”

Section: Methods and Experimental Setupmentioning

confidence: 99%

“…We continue with another possible use-cases for this collection with a reproduction of a method that uses seed studies for screening prioritisation (i.e., ranking the set of retrieved studies), see 2 in Figure 1. The techniques was originally proposed by Lee and Sun [20] and we use the reproduced implementation provided by Wang et al [34]. We investigate the effectiveness of seed-driven document ranking (SDR) methods, again comparing seed studies with pseudo seed studies.…”

Section: Screening Prioritisationmentioning

confidence: 99%

“…3 5.1.2 Multiple pseudo seed studies. Using multiple pseudo seed studies is more effectiveness then using a single seed study for SDR [34]. For our collection, we also perform SDR with multiple pseudo seed studies.…”

Section: 11mentioning

confidence: 99%

“…For our collection, we also perform SDR with multiple pseudo seed studies. We adopt the experimental settings of Wang et al [34] to evaluate the effectiveness of SDR when using multiple pseudo seed studies. We adopt the same seed study grouping strategy in which 20% of included studies are chosen using a slidingwindow approach.…”

Section: 11mentioning

confidence: 99%

“…The techniques that information specialists use to design and validate their search using seed studies have not gone unnoticed by the Information Retrieval community. There have been several studies that have exploited seed studies for automatic query formulation [29] and screening prioritisation (i.e., ranking the set of retrieved studies) [20,34]. However, one major limitation of these previous studies is that the collections that they used did not contain any seed studies.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

From Little Things Big Things Grow: A Collection with Seed Studies for Medical Systematic Review Literature Search

Wang,

Scells,

Clark

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Medical systematic review query formulation is a highly complex task done by trained information specialists. Complexity comes from the reliance on lengthy Boolean queries, which express a detailed research question. To aid query formulation, information specialists use a set of exemplar documents, called 'seed studies', prior to query formulation. Seed studies help verify the effectiveness of a query prior to the full assessment of retrieved studies. Beyond this use of seeds, specific IR methods can exploit seed studies for guiding both automatic query formulation and new retrieval models. One major limitation of work to date is that these methods exploit 'pseudo seed studies' through retrospective use of included studies (i.e., relevance assessments). However, we show pseudo seed studies are not representative of real seed studies used by information specialists. Hence, we provide a test collection with real world seed studies used to assist with the formulation of queries. To support our collection, we provide an analysis, previously not possible, on how seed studies impact retrieval and perform several experiments using seed-study based methods to compare the effectiveness of using seed studies versus pseudo seed studies. We make our test collection and the results of all of our experiments and analysis available at http://github.com/ielab/sysrev-seed-collection.

show abstract

Section: Methods and Experimental Setupmentioning

confidence: 99%

Section: Screening Prioritisationmentioning

confidence: 99%

Section: 11mentioning

confidence: 99%

Section: 11mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

From Little Things Big Things Grow: A Collection with Seed Studies for Medical Systematic Review Literature Search

Wang,

Scells,

Clark

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

A comparison of machine learning methods to find clinical trials for inclusion in new systematic reviews from their PROSPERO registrations prior to searching and screening

Liu,

Bourgeois,

Narang

et al. 2023

Research Synthesis Methods

View full text Add to dashboard Cite

Searching for trials is a key task in systematic reviews and a focus of automation. Previous approaches required knowing examples of relevant trials in advance, and most methods are focused on published trial articles. To complement existing tools, we compared methods for finding relevant trial registrations given a International Prospective Register of Systematic Reviews (PROSPERO) entry and where no relevant trials have been screened for inclusion in advance. We compared SciBERT‐based (extension of Bidirectional Encoder Representations from Transformers) PICO extraction, MetaMap, and term‐based representations using an imperfect dataset mined from 3632 PROSPERO entries connected to a subset of 65,662 trial registrations and 65,834 trial articles known to be included in systematic reviews. Performance was measured by the median rank and recall by rank of trials that were eventually included in the published systematic reviews. When ranking trial registrations relative to PROSPERO entries, 296 trial registrations needed to be screened to identify half of the relevant trials, and the best performing approach used a basic term‐based representation. When ranking trial articles relative to PROSPERO entries, 162 trial articles needed to be screened to identify half of the relevant trials, and the best‐performing approach used a term‐based representation. The results show that MetaMap and term‐based representations outperformed approaches that included PICO extraction for this use case. The results suggest that when starting with a PROSPERO entry and where no trials have been screened for inclusion, automated methods can reduce workload, but additional processes are still needed to efficiently identify trial registrations or trial articles that meet the inclusion criteria of a systematic review.

show abstract

SALτ: efficiently stopping TAR by improving priors estimates

Molinari,

Esuli

2023

Data Min Knowl Disc

View full text Add to dashboard Cite

In high recall retrieval tasks, human experts review a large pool of documents with the goal of satisfying an information need. Documents are prioritized for review through an active learning policy, and the process is usually referred to as Technology-Assisted Review (TAR). TAR tasks also aim to stop the review process once the target recall is achieved to minimize the annotation cost. In this paper, we introduce a new stopping rule called SAL$$_\tau ^R$$ τ R (SLD for Active Learning), a modified version of the Saerens–Latinne–Decaestecker algorithm (SLD) that has been adapted for use in active learning. Experiments show that our algorithm stops the review well ahead of the current state-of-the-art methods, while providing the same guarantees of achieving the target recall.

show abstract

Seed-Driven Document Ranking for Systematic Reviews: A Reproducibility Study

Cited by 9 publications

References 17 publications

From Little Things Big Things Grow: A Collection with Seed Studies for Medical Systematic Review Literature Search

From Little Things Big Things Grow: A Collection with Seed Studies for Medical Systematic Review Literature Search

A comparison of machine learning methods to find clinical trials for inclusion in new systematic reviews from their PROSPERO registrations prior to searching and screening

SALτ: efficiently stopping TAR by improving priors estimates

Contact Info

Product

Resources

About