Trialstreamer: a living, automatically updated database of clinical trial reports

Abstract

Objective Randomized controlled trials (RCTs) are the gold standard method for evaluating whether a treatment works in healthcare, but can be difficult to find and make use of. We describe the development and evaluation of a system to automatically find and categorize all new RCT reports. Materials and Methods Trialstreamer, continuously monitors PubMed and the WHO International Clinical Trials Registry Platform (ICTRP), looking for new RCTs in humans using a validated classifier. We combine machine learning and rule-based methods to extract information from the RCT abstracts, including free-text descriptions of trial populations, interventions and outcomes (the 'PICO') and map these snippets to normalised MeSH vocabulary terms. We additionally identify sample sizes, predict the risk of bias, and extract text conveying key findings. We store all extracted data in a database which we make freely available for download, and via a search portal, which allows users to enter structured clinical queries. Results are ranked automatically to prioritize larger and higher-quality studies. Results As of May 2020, we have indexed 669,895 publications of RCTs, of which 18,485 were published in the first four months of 2020 (144/day). We additionally include 303,319 trial registrations from ICTRP. The median trial sample size in the RCTs was 66. Conclusions We present an automated system for finding and categorising RCTs. This yields a novel resource: A database of structured information automatically extracted for all published RCTs in humans. We make daily updates of this database available on our website (trialstreamer.robotreviewer.net).

Data extraction methods for systematic review (semi)automation: A living systematic review

“…27 Further metrics included under 'Other' were odds ratios, 37 normalised discounted cumulative gain, 29 'sentences needed to screen per article' in order to find one relevant sentence, 38 McNemar test, 36 C-statistic (with 95% CI) and Brier score (with 95% CI). 39 Real-life evaluations, such as the percentage of outputs needing human correction, or time saved per article, were reported by one publication, 30 and an evaluation as part of a wider screening system was done in another. 40 There were several approaches and justifications of using macro-or micro-averaged precision, recall, or F1 scores in the included publications.…”

“…One, for example, applied their system to new, unlabelled data and reported that classifying the whole of PubMed takes around 20 hours using a graphics processing unit (GPU). 39 In another example, the authors reported using Google Colab GPUs, along with estimates of computing time for different training settings. 68 3.4.2.4 Is the source code available?…”

“…A small number of publications did a real-life assessment, where the data extraction algorithm was applied to different, unlabelled, and often much larger datasets or tested while conducting actual systematic reviews. 30,33,39,42,68,71,72 3.4.3.2 Are basic metrics reported (true/false positives and negatives)?…”

State of the evidence: a survey of global disparities in clinical trials

“…We developed a machine learning system which maintains a live database of annotated RCT reports, named Trialstreamer. We have described the computational methods and accuracy of the system components in detail elsewhere, 11 and summarise the key points relevant to the current study below.…”

“… 14 These machine learning models are trained 1 on 280 000 abstracts manually labelled as being RCTs or not by Cochrane Crowd, ( https://crowd.cochrane.org ) a collaborative citizen science project. We do not rely on the Publication Type index alone, as we have previously found it to miss a substantial proportion of the 5–7 most recent years of articles 11 (due to delay in manual indexing after publication). We next removed any RCTs that were not conducted in humans (eg, animal or agricultural studies) using an SVM model.…”