Using a Secure, Continually Updating, Web Source Processing Pipeline to Support the Real-Time Data Synthesis and Analysis of Scientific Literature: Development and Validation Study

Abstract

Background The scale and quality of the global scientific response to the COVID-19 pandemic have unquestionably saved lives. However, the COVID-19 pandemic has also triggered an unprecedented “infodemic”; the velocity and volume of data production have overwhelmed many key stakeholders such as clinicians and policy makers, as they have been unable to process structured and unstructured data for evidence-based decision making. Solutions that aim to alleviate this data synthesis–related challenge are unable to capture heterogeneous web data in real time for the production of concomitant answers and are not based on the high-quality information in responses to a free-text query. Objective The main objective of this project is to build a generic, real-time, continuously updating curation platform that can support the data synthesis and analysis of a scientific literature framework. Our secondary objective is to validate this platform and the curation methodology for COVID-19–related medical literature by expanding the COVID-19 Open Research Dataset via the addition of new, unstructured data. Methods To create an infrastructure that addresses our objectives, the PanSurg Collaborative at Imperial College London has developed a unique data pipeline based on a web crawler extraction methodology. This data pipeline uses a novel curation methodology that adopts a human-in-the-loop approach for the characterization of quality, relevance, and key evidence across a range of scientific literature sources. Results REDASA (Realtime Data Synthesis and Analysis) is now one of the world’s largest and most up-to-date sources of COVID-19–related evidence; it consists of 104,000 documents. By capturing curators’ critical appraisal methodologies through the discrete labeling and rating of information, REDASA rapidly developed a foundational, pooled, data science data set of over 1400 articles in under 2 weeks. These articles provide COVID-19–related information and represent around 10% of all papers about COVID-19. Conclusions This data set can act as ground truth for the future implementation of a live, automated systematic review. The three benefits of REDASA’s design are as follows: (1) it adopts a user-friendly, human-in-the-loop methodology by embedding an efficient, user-friendly curation platform into a natural language processing search engine; (2) it provides a curated data set in the JavaScript Object Notation format for experienced academic reviewers’ critical appraisal choices and decision-making methodologies; and (3) due to the wide scope and depth of its web crawling method, REDASA has already captured one of the world’s largest COVID-19–related data corpora for searches and curation.

Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed

“…Eleven papers used a plethora of text-mining tools to aid search syntax building, such as Anne O’Tate, AntConc, Apache Lucene, BiblioShiny, Carrot2, CitNetExplorer, EndNote, Keyword‐Analyzer, Leximancer, Lingo3G, Lingo4G, MeSH on Demand, MetaMap, Microsoft Academic, PubReMiner, Systematic Review Accelerator, TerMine, Text Analyzer, Tm for R, VOSviewer, Voyant, Yale MeSH Analyzer, and in-house solutions [ 18 , 33 – 35 , 37 , 41 , 46 , 47 , 49 – 51 ]. Two papers introduced curated article collections, such as Cochrane CENTRAL [ 44 ], and the Realtime Data Synthesis and Analysis (REDASA) COVID-19 dataset [ 48 ], which were assembled using various automation techniques. Other tools included an automated extension of PubMed searches to the ClinicalTrials.gov database [ 40 ], a Boolean query refiner [ 42 ], a support vector machine (SVM) classifier as alternative to PubMed search filters for review updating [ 38 ], a strategy using the Patient, Intervention, Comparator, and Outcome framework (PICO) terms in the title field only [ 39 ], an automated full-text retrieval and targeted search replacing database screening [ 45 ], and a Microsoft Excel-based convenience tool to build Boolean queries [ 43 ].…”

“…Further approaches included PECO tagging in a rapid evidence mapping study using SWIFT Review [ 125 ], extraction of geographic locations from the manuscript [ 141 ], extraction of endpoints as comparative claim sentences [ 142 ], data extraction from ClinicalTrials.gov for meta-analyses [ 143 ], and convenience tools to highlight relevant sentences [ 74 ], or extract data from graphs [ 144 ]. Finally, development of the REDASA COVID-19 dataset involved human experts in the loop, web-crawling, and a natural language processing search engine to provide a real-time curated open dataset for evidence syntheses to aid pandemic response [ 48 ].…”

Automation of systematic reviews of biomedical literature: a systematic review of studies indexed in PubMed

“…Nineteen included papers (15.4%) aimed to automate or improve database searches 18,43,51,54,56,61,70,76,78,98,101,104,111,113,115,122,127,136,140 . The rst included paper from 2011 applied text-mining to construct a search syntax for PubMed, using the Apache Lucene platform 43 .…”

“…Further approaches included PECO tagging in a rapid evidence mapping study using SWIFT Review 91 , extraction of geographic locations from the manuscript 133 , extraction of endpoints as comparative claim sentences 55 , data extraction from ClinicalTrials.gov for meta-analyses 97 , and convenience tools to highlight relevant sentences 130 , or extract data from graphs 88 . Finally, development of the REDASA Covid-19 dataset involved human experts in the loop, web-crawling and a natural language processing search engine to provide a real-time curated open dataset for evidence syntheses to aid pandemic response 127 .…”

“…Some studies combined two stages from either search, screening, full-text selection or data extraction. These studies included two ASLRs 113,132 , studies on alternative SLR approaches, such as Rapid Evidence Mapping 91 and Potential Technologies Review 98 , and the REDASA Covid-19 dataset 127 . A study used automated record screening before evaluating a text mining algorithm for full text selection 108 , and another automated record screening in connection with PICO named entry recognition for data extraction 149 .…”