Systematic review toolbox

Marshall, Christopher J.; Brereton, Pearl

doi:10.1145/2745802.2745824

Cited by 52 publications

(34 citation statements)

References 3 publications

(3 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Candidate tools were gathered from the “Systematic Review Toolbox” website, which is a highly comprehensive list of tools compiled by researchers in the field. Based on publicly available information and personal experience, tools that automated any part of the systematic review process were selected.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Usage of automation tools in systematic reviews

Altena

Spijker

Olabarriaga

2019

Research Synthesis Methods

View full text Add to dashboard Cite

Systematic reviews are a cornerstone of today's evidence‐informed decision making. With the rapid expansion of questions to be addressed and scientific information produced, there is a growing workload on reviewers, making the current practice unsustainable without the aid of automation tools. While many automation tools have been developed and are available, uptake seems to be lagging. For this reason, we set out to investigate the current level of uptake and what the potential barriers and facilitators are for the adoption of automation tools in systematic reviews. We deployed surveys among systematic reviewers that gathered information on tool uptake, demographics, systematic review characteristics, and barriers and facilitators for uptake. Systematic reviewers from multiple domains were targeted during recruitment; however, responders were predominantly from the biomedical sciences. We found that automation tools are currently not widely used among the participants. When tools are used, participants mostly learn about them from their environment, for example, through colleagues, peers, or organization. Tools are often chosen on the basis of user experience, either by own experience or from colleagues or peers. Lastly, licensing, steep learning curve, lack of support, and mismatch to workflow are often reported by participants as relevant barriers. While conclusions can only be drawn for the biomedical field, our work provides evidence and confirms the conclusions and recommendations of previous work, which was based on expert opinions. Furthermore, our study highlights the importance that organizations and best practices in a field can have for the uptake of automation tools for systematic reviews.

show abstract

Section: Methodsmentioning

confidence: 99%

“…An analysis by Tsafnat et al showed that several automation tools have been developed to assist the screening of titles and abstracts based on machine learning methods. A comprehensive list can be found in the Systematic Review Toolbox and an overview of their performance can be found in O'Mara‐Eves et al and Shemilt et al…”

Section: Introductionmentioning

confidence: 99%

Usage of automation tools in systematic reviews

Altena

Spijker

Olabarriaga

2019

Research Synthesis Methods

View full text Add to dashboard Cite

show abstract

“…They highlight that low levels of funding, as well as the complexity of integrating tools for different systematic reviewing tasks have led to many small and isolated pieces of software. A working group formed at the ICASR 2019 Hackathon is compiling an overview of tools published on the Systematic Review Toolbox website 7 . This ongoing work is focused on assessing maintenance status, accessibility and supported reviewing tasks of 120 tools that can be used in any part of the systematic reviewing process as of November 2019.…”

Section: Related Researchmentioning

confidence: 99%

“…In a preliminary test our search strategy was able to identify 4320 Medline records, including all Medline-indexed records included by O'Mara-Eves et al 3 . We plan to search the Systematic Review Toolbox website for further information on any published or unpublished tools 7 .…”

Section: Search and Updatesmentioning

confidence: 99%

Data extraction methods for systematic review (semi)automation: A living review protocol

et al. 2020

View full text Add to dashboard Cite

Background: Researchers in evidence-based medicine cannot keep up with the amounts of both old and newly published primary research articles. Conducting and updating of systematic reviews is time-consuming. In practice, data extraction is one of the most complex tasks in this process. Exponential improvements in computational processing speed and data storage are fostering the development of data extraction models and algorithms. This, in combination with quicker pathways to publication, led to a large landscape of tools and methods for data extraction tasks. Objective: To review published methods and tools for data extraction to (semi)automate the systematic reviewing process. Methods: We propose to conduct a living review. With this methodology we aim to do monthly search updates, as well as bi-annual review updates if new evidence permits it. In a cross-sectional analysis we will extract methodological characteristics and assess the quality of reporting in our included papers. Conclusions: We aim to increase transparency in the reporting and assessment of machine learning technologies to the benefit of data scientists, systematic reviewers and funders of health research. This living review will help to reduce duplicate efforts by data scientists who develop data extraction methods. It will also serve to inform systematic reviewers about possibilities to support their data extraction.

show abstract

“…Although many machine learning tools exist to expedite screening, 16 we chose Abstrackr, DistillerSR, and RobotAnalyst for the following reasons. First, the development of these tools has been well documented.…”

Section: Machine Learning Toolsmentioning

confidence: 99%

Performance and Usability of Machine Learning for Screening in Systematic Reviews: A Comparative Evaluation of Three Tools

Gates¹,

Guitard²,

Pillay³

et al. 2019

View full text Add to dashboard Cite

Purpose of project For title and abstract screening, we explored the reliability of three machine learning tools when used to automatically eliminate irrelevant records or complement the work of a single reviewer. We evaluated the usability of each tool. Key messages • The reliability of the tools to automatically eliminate irrelevant records was highly variable; a median (range) 70% (0-100%) of relevant records were missed compared to dual independent screening. • Abstrackr and RobotAnalyst improved upon single reviewer screening by identifying studies that the single reviewer missed, but performance was not reliable. DistillerSR provided no advantage over single reviewer screening. • The tools' usability relied on multiple properties: user friendliness; qualities of the user interface; features and functions; trustworthiness; ease and speed of obtaining the predictions; and practicality of the export files. • Standards for conducting and reporting evaluations of machine learning tools for screening will facilitate their replication. This report is based on research conducted by the University of Alberta Evidence-based Practice Center under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. 290-2015-00001-I). The findings and conclusions in this document are those of the authors, who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services. None of the investigators have any affiliations or financial involvement that conflicts with the material presented in this report. The information in this report is intended to help healthcare decision makers-patients and clinicians, health system leaders, and policymakers, among others-make well-informed decisions and thereby improve the quality of healthcare services. This report is not intended to be a substitute for the application of clinical judgment. Anyone who makes decisions concerning the provision of clinical care should consider this report in the same way as any medical reference and in conjunction with all other pertinent information, i.e., in the context of available resources and circumstances presented by individual patients.

show abstract

Systematic review toolbox

Cited by 52 publications

References 3 publications

Usage of automation tools in systematic reviews

Usage of automation tools in systematic reviews

Data extraction methods for systematic review (semi)automation: A living review protocol

Performance and Usability of Machine Learning for Screening in Systematic Reviews: A Comparative Evaluation of Three Tools

Contact Info

Product

Resources

About