Living systematic reviews: 2. Combining human and machine effort

Thomas, James; Noel-Storr, Anna; Marshall, Iain J.; Wallace, Byron C.; McDonald, Steve; Mavergames, Chris; Glasziou, Paul; Shemilt, Ian; Synnot, Anneliese; Turner, Tari; Elliott, Julian

doi:10.1016/j.jclinepi.2017.08.011

Cited by 264 publications

(133 citation statements)

References 12 publications

Supporting

Mentioning

124

Contrasting

Unclassified

Order By: Relevance

“…However, recent promising preliminary work has found that crowdsourced approaches can yield surprisingly high-quality annotations in the domain of EBM specifically (Mortensen et al, 2017; Thomas et al, 2017; Wallace et al, 2017). …”

Section: Related Workmentioning

confidence: 99%

A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature

Nye¹,

Li²,

Patel³

et al. 2018

Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Self Cite

133

195

View full text Add to dashboard Cite

We present a corpus of 5,000 richly annotated abstracts of medical articles describing clinical randomized controlled trials. Annotations include demarcations of text spans that describe the Patient population enrolled, the Interventions studied and to what they were Compared, and the Outcomes measured (the ‘PICO’ elements). These spans are further annotated at a more granular level, e.g., individual interventions within them are marked and mapped onto a structured medical vocabulary. We acquired annotations from a diverse set of workers with varying levels of expertise and cost. We describe our data collection process and the corpus itself in detail. We then outline a set of challenging NLP tasks that would aid searching of the medical literature and the practice of evidence-based medicine.

show abstract

Section: Related Workmentioning

confidence: 99%

A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature

Nye¹,

Li²,

Patel³

et al. 2018

Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Self Cite

133

195

View full text Add to dashboard Cite

show abstract

“…With prioritisation, it may be possible to perform a partial screening and still identify the vast majority, for example, 95%, of the relevant references . Prioritisation may also enable different screening or review paradigms, such as living systematic reviews or updates, since more relevant references are found earlier than with manual screening …”

Section: Introductionmentioning

confidence: 99%

Prioritising references for systematic reviews with RobotAnalyst: A user study

Przybyła

Brockmeier

Kontonatsios

et al. 2018

Research Synthesis Methods

View full text Add to dashboard Cite

Screening references is a time‐consuming step necessary for systematic reviews and guideline development. Previous studies have shown that human effort can be reduced by using machine learning software to prioritise large reference collections such that most of the relevant references are identified before screening is completed. We describe and evaluate RobotAnalyst, a Web‐based software system that combines text‐mining and machine learning algorithms for organising references by their content and actively prioritising them based on a relevancy classification model trained and updated throughout the process. We report an evaluation over 22 reference collections (most are related to public health topics) screened using RobotAnalyst with a total of 43 610 abstract‐level decisions. The number of references that needed to be screened to identify 95% of the abstract‐level inclusions for the evidence review was reduced on 19 of the 22 collections. Significant gains over random sampling were achieved for all reviews conducted with active prioritisation, as compared with only two of five when prioritisation was not used. RobotAnalyst's descriptive clustering and topic modelling functionalities were also evaluated by public health analysts. Descriptive clustering provided more coherent organisation than topic modelling, and the content of the clusters was apparent to the users across a varying number of clusters. This is the first large‐scale study using technology‐assisted screening to perform new reviews, and the positive results provide empirical evidence that RobotAnalyst can accelerate the identification of relevant studies. The results also highlight the issue of user complacency and the need for a stopping criterion to realise the work savings.

show abstract

“…One solution to accelerate the SR process and make it more efficient is the use of computer technology during the process . The utilization of technology to automate the SR process is not new, but with advances in technology, a series of articles were recently published discussing the importance of automation of SRs and presenting tools to assist with this process . Tsafnat et al posited that the use of technology can help clinicians such that they always have access to the best evidence available .…”

Section: Resultsmentioning

confidence: 99%

“…Tsafnat et al posited that the use of technology can help clinicians such that they always have access to the best evidence available . A recent article highlighted that combining human and machine efforts can render the process more efficient in terms of review tasks, such as team formation, search, eligibility assessment, data extraction, and collection as well as synthesis …”

Section: Resultsmentioning

confidence: 99%

Systematic reviews in restorative dentistry: discussing relevant aspects

Sarkis‐Onofre

Pereira‐Cenci

Tricco

et al. 2019

J Esthet Restor Dent

View full text Add to dashboard Cite

Objective This article aims to discuss key aspects of systematic reviews (SR) focusing on the improvement of the conduct and reporting. Methods Important aspects of SRs, such as prospective registration of the review protocol, basic structure, inclusion criteria, use of the Preferred Reporting Items for Systematic Reviews and Meta‐Analyses (PRISMA) Statement, confidence in the results and future directions are discussed. To determine relevant aspects, a search was conducted without date limitations in PubMed (October 15th, 2017) to identify SRs written in English evaluating clinical performance of direct composite resin restoration in permanent posterior teeth or comparing direct composite resin with other material/techniques. The quality of SRs included was assessed using the Assessing the Methodological Quality of Systematic Reviews 2 tool. Results Fifteen SRs were included. The overall confidence in the results of SRs was classified as critically low. Some aspects should be highlighted: SRs of in vitro studies are an important tool in restorative dentistry, and initiatives such as the PRISMA Statement and PROSPERO should be considered a standard code of practice. Conclusions The compliance with and awareness of the discussed aspects may be a significant feature of the improvement of SR quality in the dentistry. Clinical significance Initiatives such as the PRISMA Statement and PROSPERO should be taken in account by systematic reviewers in dentistry to improve the conduct and reporting of SRs, and to make their reviews are more clinically helpful.

show abstract

Living systematic reviews: 2. Combining human and machine effort

Cited by 264 publications

References 12 publications

A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature

A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature

Prioritising references for systematic reviews with RobotAnalyst: A user study

Systematic reviews in restorative dentistry: discussing relevant aspects

Contact Info

Product

Resources

About