Data extraction methods for systematic review (semi)automation: A living review protocol

Schmidt, Lena; Olorisade, Babatunde Kazeem; McGuinness, Luke A; Higgins, Julian P T

doi:10.12688/f1000research.22781.1

Cited by 6 publications

(21 citation statements)

References 11 publications

(13 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The process of updating started as described in the protocol 11 and is shown in Figure 1. In short, we will continuously update the literature search results, using the search strategies and methods described in the section 'Search' below.…”

Section: Living Review Methodologymentioning

confidence: 99%

See 1 more Smart Citation

Data extraction methods for systematic review (semi)automation: A living systematic review

et al. 2021

Self Cite

View full text Add to dashboard Cite

Background: The reliable and usable (semi)automation of data extraction can support the field of systematic review by reducing the workload required to gather information about the conduct and results of the included studies. This living systematic review examines published approaches for data extraction from reports of clinical studies. Methods: We systematically and continually search MEDLINE, Institute of Electrical and Electronics Engineers (IEEE), arXiv, and the dblp computer science bibliography databases. Full text screening and data extraction are conducted within an open-source living systematic review application created for the purpose of this review. This iteration of the living review includes publications up to a cut-off date of 22 April 2020. Results: In total, 53 publications are included in this version of our review. Of these, 41 (77%) of the publications addressed extraction of data from abstracts, while 14 (26%) used full texts. A total of 48 (90%) publications developed and evaluated classifiers that used randomised controlled trials as the main target texts. Over 30 entities were extracted, with PICOs (population, intervention, comparator, outcome) being the most frequently extracted. A description of their datasets was provided by 49 publications (94%), but only seven (13%) made the data publicly available. Code was made available by 10 (19%) publications, and five (9%) implemented publicly available tools. Conclusions: This living systematic review presents an overview of (semi)automated data-extraction literature of interest to different types of systematic review. We identified a broad evidence base of publications describing data extraction for interventional reviews and a small number of publications extracting epidemiological or diagnostic accuracy data. The lack of publicly available gold-standard data for evaluation, and lack of application thereof, makes it difficult to draw conclusions on which is the best-performing system for each data extraction target. With this living review we aim to review the literature continually.

show abstract

Section: Living Review Methodologymentioning

confidence: 99%

“…This review was conducted following a preregistered and published protocol. 10,11 PROSPERO was initially considered as platform for registration, but it is limited to reviews with health-related outcomes. Any deviations from the protocol have been described below.…”

Section: Registration/protocolmentioning

confidence: 99%

Data extraction methods for systematic review (semi)automation: A living systematic review

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…The first cluster includes computational aspects such as data mining, while the second cluster identifies publication related to systematic reviews. The Medline search strategy is provided as Extended data 11 . We aim to adapt this search strategy for conducting searches in all mentioned databases.…”

Section: Search and Updatesmentioning

confidence: 99%

“…• We will include full text publications that describe an original natural language processing approach to extract data related to systematic reviewing tasks. Data fields of interest are adapted from the Cochrane Handbook for Systematic Reviews of Interventions 13 , and defined in the Extended data 11 . We will include the full range of natural language processing (NLP) methods, including for example regular expressions, rule-based systems, machine learning, and deep neural networks.…”

Section: Eligibility Criteria Eligible Papersmentioning

confidence: 99%

See 1 more Smart Citation

Data extraction methods for systematic review (semi)automation: A living review protocol

et al. 2020

Self Cite

View full text Add to dashboard Cite

Background: Researchers in evidence-based medicine cannot keep up with the amounts of both old and newly published primary research articles. Support for the early stages of the systematic review process – searching and screening studies for eligibility – is necessary because it is currently impossible to search for relevant research with precision. Better automated data extraction may not only facilitate the stage of review traditionally labelled ‘data extraction’, but also change earlier phases of the review process by making it possible to identify relevant research. Exponential improvements in computational processing speed and data storage are fostering the development of data mining models and algorithms. This, in combination with quicker pathways to publication, led to a large landscape of tools and methods for data mining and extraction. Objective: To review published methods and tools for data extraction to (semi)automate the systematic reviewing process. Methods: We propose to conduct a living review. With this methodology we aim to do constant evidence surveillance, bi-monthly search updates, as well as review updates every 6 months if new evidence permits it. In a cross-sectional analysis we will extract methodological characteristics and assess the quality of reporting in our included papers. Conclusions: We aim to increase transparency in the reporting and assessment of automation technologies to the benefit of data scientists, systematic reviewers and funders of health research. This living review will help to reduce duplicate efforts by data scientists who develop data mining methods. It will also serve to inform systematic reviewers about possibilities to support their data extraction.

show abstract

Evaluating the Efficacy of Large Language Models for Systematic Review and Meta-Analysis Screening

Luo,

Sastimoglu,

Faisal

et al. 2024

Preprint

View full text Add to dashboard Cite

BackgroundSystematic reviews and meta-analyses are essential for informed research and policymaking, yet they are typically resource-intensive and time-consuming. Recent advances in artificial intelligence and machine learning offer promising opportunities to streamline these processes.ObjectiveTo enhance the efficiency of systematic reviews, we explored the automation of various stages using GPT-3.5 Turbo. We assessed the model’s efficacy and performance by comparing it against three expert-conducted reviews across a comprehensive dataset of 24,534 studies.MethodsThe model’s performance was evaluated through a comparison with three expert reviews, utilizing a pseudo-K-folds permutation and a one-tailed ANOVA with an alpha level of 0.05 to ensure statistical validity. Key performance metrics such as accuracy, sensitivity, specificity, predictive values, F1-score, and the Matthews correlation coefficient were analyzed using two sets of prompts.ResultsOur approach significantly streamlined the systematic review process, which typically takes a year, reducing it to a few hours without sacrificing quality. In the initial screening phase, accuracy, specificity, and negative predictive values ranged between 80% and 95%. Sensitivity improved markedly during the second screening phase, demonstrating the model’s robustness when provided with more extensive data.ConclusionWhile ongoing refinements are needed, this tool represents a significant advancement in research methodologies, potentially making systematic reviews more accessible to a wider range of researchers.Impact StatementOur manuscript presents a novel review screening protocol built using open-source frameworks, which significantly enhances the systematic review process in terms of efficiency and cost-effectiveness. Leveraging the capabilities of GPT and embedding models, our protocol demonstrates the potential to transform a traditionally time-consuming and expensive task into an accelerated and economical operation, all while maintaining high standards of accuracy and reliability.Key PointsGPT screening can streamline systematic reviews from a year-long, expensive process to just hours at minimal cost.Validated across different topics, the protocol exhibits high reliability and consistency in study inclusion.The AI-driven process reduces human bias, with prompt optimization considerably improving sensitivity.

show abstract

Data extraction methods for systematic review (semi)automation: A living review protocol

Cited by 6 publications

References 11 publications

Data extraction methods for systematic review (semi)automation: A living systematic review

Data extraction methods for systematic review (semi)automation: A living systematic review

Data extraction methods for systematic review (semi)automation: A living review protocol

Evaluating the Efficacy of Large Language Models for Systematic Review and Meta-Analysis Screening

Contact Info

Product

Resources

About