A Test Collection for Evaluating Retrieval of Studies for Inclusion in Systematic Reviews

Abstract

This version is available at https://strathprints.strath.ac.uk/62696/ Strathprints is designed to allow users to access the research output of the University of Strathclyde. Unless otherwise explicitly stated on the manuscript, Copyright © and Moral Rights for the papers on this site are retained by the individual authors and/or other copyright owners. Please check the manuscript for details of any other licences that may have been applied. You may not engage in further distribution of the material for any profitmaking activities or any commercial gain. You may freely distribute both the url (https://strathprints.strath.ac.uk/) and the content of this paper for research or private study, educational, or not-for-profit purposes without prior permission or charge.Any correspondence concerning this service should be sent to the Strathprints administrator: strathprints@strath.ac.ukThe Strathprints institutional repository (https://strathprints.strath.ac.uk) is a digital archive of University of Strathclyde research outputs. It has been developed to disseminate open access research outputs, expose data about those outputs, and enable the management and persistent access to Strathclyde's intellectual output. A Test Collection for Evaluating Retrieval of Studies for Inclusion in Systematic ReviewsHarrisen Scells ABSTRACT is paper introduces a test collection for evaluating the e ectiveness of di erent methods used to retrieve research studies for inclusion in systematic reviews. Systematic reviews appraise and synthesise studies that meet speci c inclusion criteria. Systematic reviews intended for a biomedical science audience use boolean queries with many, o en complex, search clauses to retrieve studies; these are then manually screened to determine eligibility for inclusion in the review. is process is expensive and time consuming. e development of systems that improve retrieval e ectiveness will have an immediate impact by reducing the complexity and resources required for this process. Our test collection consists of approximately 26 million research studies extracted from the freely available MEDLINE database, 94 review (query) topics extracted from Cochrane systematic reviews, and corresponding relevance assessments. Tasks for which the collection can be used for information retrieval system evaluation are described and the use of the collection to evaluate common baselines within one such task is demonstrated. e test collection is available at h ps://github.com/ielab/SIGIR2017-PICO-Collection.

A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature

“…Larger corpora for EBM tasks have been derived using (noisy) automated annotation approaches. This approach has been used to build, e.g., datasets to facilitate work on Information Retrieval (IR) models for biomedical texts (Scells et al, 2017;Chung, 2009;Boudin et al, 2010). Similar approaches have been used to 'distantly supervise' annotation of full-text articles describing clinical trials .…”

A Dataset of Systematic Review Updates

“…Rece♪tly the CLEF eHealth track o♪ Tech♪ology Assisted Reviews i♪ Empirical ℧edici♪e [9,20] developed datasets co♪tai♪i♪g 72 topics created from diag♪ostic test accuracy systematic reviews produced by the Cochra♪e Col-laboratio♪. A♪other test collectio♪ has also bee♪ derived from 94 Cochra♪e reviews [18]. However, ♪o♪e of these datasets focus o♪ the review updates.…”

Seed-driven document ranking for systematic reviews in evidence-based medicine

“…There is a recently published dataset for evaluating the retrieval process in systematic reviews [7]. However, this dataset does not contain information (e.g., pmids) of all candidate documents of its SRs.…”

“…In addition, we evaluate a ranking model based on word embeddings, which we call average embedding similarity (AES) 7 . Word embedding is an unsupervised approach to learn continuous distributed vector representations of words.…”

“…It is difficult to train neural models specialized for individual SRs because of too few relevant documents in a SR. We use train and test splits in CLEF19, consisting of 72 and 7 SRs, respectively 7 . For a development (dev) dataset, we randomly sample 7 SRs in the training 72 SRs.…”