Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools

Gates, Allison; Guitard, Samantha; Pillay, Jennifer; Elliott, Sarah A.; Dyson, Michele P; Newton, Amanda S; Hartling, Lisa

doi:10.1186/s13643-019-1222-2

Cited by 69 publications

(108 citation statements)

References 28 publications

(32 reference statements)

Supporting

Mentioning

103

Contrasting

Unclassified

Order By: Relevance

“…We used Abstrackr (http://abstrackr.cebm.brown.edu) [22], an online ML tool for title-abstract screening, for this study. Among the many available tools, we chose Abstrackr because it is freely-available and testing at our centre found it to be more reliable and user friendly than other available tools [10].…”

Section: Machine Learning Tool: Abstrackrmentioning

confidence: 99%

“…Because our evaluation was retrospective, we estimated time savings based on a screening rate of two records per minute. Although ambitious, this rate allowed for more conservative estimates of time savings and for comparisons to previous studies that have used the same rate [10,15,16].…”

Section: Strengths and Limitationsmentioning

confidence: 99%

“…In spite of the clear need to create e ciencies in systematic review production [1,2] and the accrual of evidence highlighting the bene ts and risks [8,9], and usability [10] of available ML tools, the adoption of ML-assisted methods has been slow [8,11,12]. In a 2019 commentary, O'Connor et al summarized possible barriers to adoption, including distrust by review teams and end users of systematic reviews; setup challenges and incompatibility with traditional work ows; and inadequate awareness of available tools [13].…”

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…Previously published studies undertaken at our evidence synthesis centre [10,15,16] have addressed the bene ts (workload and estimated time savings) and risks (omitting relevant studies) of various MLassisted screening approaches in systematic and rapid reviews. We have also explored the usability of some available ML tools [10]. Despite promising ndings, in the absence of clear guidance or endorsement by evidence synthesis organizations, it remains unclear how ML-assisted methods should (or could) be incorporated into practice [13].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Decoding semi-automated title-abstract screening: a retrospective exploration of the review, study, and publication characteristics associated with accurate relevance predictions

Gates

DaRosa

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Abstract Background We evaluated the benefits and risks of using the Abstrackr machine learning (ML) tool to semi-automate title-abstract screening, and explored whether Abstrackr’s predictions varied by review or study-level characteristics. Methods For 16 reviews we screened a 200-record training set in Abstrackr and downloaded the predicted relevance of the remaining records. We retrospectively simulated the liberal-accelerated screening approach: one reviewer screened the records predicted as relevant; a second reviewer screened those predicted as irrelevant and those excluded by the first reviewer. We estimated the time savings and proportion missed compared with dual independent screening. For reviews with pairwise meta-analyses, we evaluated changes to the pooled effects after removing the missed studies. We explored whether the tool’s predictions varied by review and study-level characteristics using Fisher’s Exact and unpaired t-tests. Results Using the ML-assisted liberal-accelerated approach, we wrongly excluded 0 to 3 (0 to 14%) records but saved a median (IQR) 26 (33) hours of screening time. Removing missed studies from meta-analyses did not alter the reviews’ conclusions. Of 802 records in the final reports, 87% were correctly predicted as relevant. The correctness of the predictions did not differ by review (systematic or rapid, P=0.37) or intervention type (simple or complex, P=0.47). The predictions were more often correct in reviews with multiple (89%) vs. single (83%) research questions (P=0.01), or that included only trials (95%) vs. multiple designs (86%) (P=0.003). At the study level, trials (91%), mixed methods (100%), and qualitative (93%) studies were more often correctly predicted as relevant compared with observational studies (79%) or reviews (83%) (P=0.0006). Studies at high or unclear (88%) vs. low risk of bias (80%) (P=0.039), and those published more recently (mean (SD) 2008 (7) vs. 2006 (10), P=0.02) were more often correctly predicted as relevant. Conclusion Our screening approach saved time and may be suitable in conditions where the limited risk of missing relevant records is acceptable. ML-assisted screening may be most trustworthy for reviews that seek to include only trials. Several of our findings are paradoxical, and require further study to fully understand the tasks to which ML-assisted screening is best suited.

show abstract

Section: Machine Learning Tool: Abstrackrmentioning

confidence: 99%

Section: Strengths and Limitationsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Decoding semi-automated title-abstract screening: a retrospective exploration of the review, study, and publication characteristics associated with accurate relevance predictions

Gates

DaRosa

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Healthcare workers’ informal uses of mobile devices to support their work: a qualitative evidence synthesis

Glenton

Paulsen

Agarwal

et al. 2023

Cochrane Database of Systematic Reviews

View full text Add to dashboard Cite

Machine learning for accelerating screening in evidence reviews

Chappell

Edwards

Watkins

et al. 2023

Cochrane Evidence Synthesis and Methods

View full text Add to dashboard Cite

Evidence reviews are important for informing decision-making and primary research, but they can be time-consuming and costly. With the advent of artificial intelligence, including machine learning, there is an opportunity to accelerate the review process at many stages, with study screening identified as a prime candidate for assistance.Despite the availability of a large number of tools promising to assist with study screening, these are not consistently used in practice and there is skepticism about their application. Single-arm evaluations suggest the potential for tools to reduce screening burden. However, their integration into practice may need further investigation through evaluations of outcomes such as overall resource use and impact on review findings and recommendations. Because the literature lacks comparative studies, it is not currently possible to determine their relative accuracy.In this commentary, we outline the published research and discuss options for incorporating tools into the review workflow, considering the needs and requirements of different types of review.

show abstract

Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools

Cited by 69 publications

References 28 publications

Decoding semi-automated title-abstract screening: a retrospective exploration of the review, study, and publication characteristics associated with accurate relevance predictions

Decoding semi-automated title-abstract screening: a retrospective exploration of the review, study, and publication characteristics associated with accurate relevance predictions

Healthcare workers’ informal uses of mobile devices to support their work: a qualitative evidence synthesis

Machine learning for accelerating screening in evidence reviews

Contact Info

Product

Resources

About