Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error

Abstract

BackgroundHere we outline a method of applying existing machine learning (ML) approaches to aid citation screening in an on-going broad and shallow systematic review of preclinical animal studies, with the aim of achieving a high performing algorithm comparable to human screening.MethodsWe applied ML approaches to a broad systematic review of animal models of depression at the citation screening stage. We tested two independently developed ML approaches which used different classification models and feature sets. We recorded the performance of the ML approaches on an unseen validation set of papers using sensitivity, specificity and accuracy. We aimed to achieve 95% sensitivity and to maximise specificity. The classification model providing the most accurate predictions was applied to the remaining unseen records in the dataset and will be used in the next stage of the preclinical biomedical sciences systematic review. We used a cross validation technique to assign ML inclusion likelihood scores to the human screened records, to identify potential errors made during the human screening process (error analysis).ResultsML approaches reached 98.7% sensitivity based on learning from a training set of 5749 records, with an inclusion prevalence of 13.2%. The highest level of specificity reached was 86%. Performance was assessed on an independent validation dataset. Human errors in the training and validation sets were successfully identified using assigned the inclusion likelihood from the ML model to highlight discrepancies. Training the ML algorithm on the corrected dataset improved the specificity of the algorithm without compromising sensitivity. Error analysis correction leads to a 3% improvement in sensitivity and specificity, which increases precision and accuracy of the ML algorithm.ConclusionsThis work has confirmed the performance and application of ML algorithms for screening in systematic reviews of preclinical animal studies. It has highlighted the novel use of ML algorithms to identify human error. This needs to be confirmed in other reviews, , but represents a promising approach to integrating human decisions and automation in systematic review methodology.

Identifying and understanding factors that affect the translation of therapies from the laboratory to patients: a study protocol

“…If an adequate sample size of 40 cases is not reached, we will continue searching for promising preclinical therapies in order to identify additional phase II or later translated therapies. The CAMARADES group has used a machine learning algorithm 35 to identify a cohort of over 100,000 reports of in vivo research available on PubMed Central. From this we will randomly select reports of in vivo preclinical research and assess these for clinical translation.…”

Learning from Principles of Evidence-Based Medicine to Optimize Nonclinical Research Practices

“…Meta-analyses are often performed in collaborations, and a recent feasibility study using crowd-sourcing for clinical study quality assessment suggests that this could be a way forward, since experts and novices obtained the same results (Pianta et al 2018). Combined with recently developed and highly promising machine learning algorithms (Bannach-Brown et al 2019), collaborative efforts could increase the pace and reduce human error in systematic reviews and meta-analysis.…”

Statistical Stopping Criteria for Automated Screening in Systematic Reviews

“…This is particularly problematic in systematic reviews because low recall increases the risk of bias [9]. The lack of appropriate stopping criteria has therefore been identified as a research gap [10,11], although some approaches have been suggested. These have most commonly fallen into the following categories:…”