Machine learning reduced workload for the Cochrane COVID-19 Study Register: development and evaluation of the Cochrane COVID-19 Study Classifier

Shemilt, Ian; Noel-Storr, Anna; Thomas, James; Featherstone, Robin; Mavergames, Chris

doi:10.1186/s13643-021-01880-6

Cited by 19 publications

(8 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast, initiatives with broader scope, for example registers of COVID-19 research, faced challenges that were addressed through automation, specifically machine learning classifiers [23]. Minimizing screening through automation was also implemented when researchers faced more heterogeneous studies and data, as was the case with the suite of diagnostic test reviews published by Cochrane, or where evidence was monitored across a broad portfolio, as was the case with the living guidelines maintained by NICE [24].…”

Section: Discussionmentioning

confidence: 99%

Designing, refining and reflecting on 3 years of daily evidence surveillance for Australia's living national COVID‐19 guidelines

McDonald,

Turner

2024

Cochrane Evidence Synthesis and Methods

View full text Add to dashboard Cite

Australia's national living guidelines for COVID‐19 were launched in April 2020 and include over 200 recommendations to guide the clinical care of patients with COVID‐19. Until the guidelines were retired in August 2023, new evidence was continually monitored through daily surveillance. This paper describes the initial design for evidence surveillance (at a time when efforts to collate information on the novel coronavirus were in their infancy and often duplicative) and how it was refined throughout the pandemic. Among the wide range of sources monitored, the collections of COVID‐19 research from leading medical journals, medRxiv preprints and PubMed auto alerts proved the most enduring in identifying time‐critical and impactful evidence. The paper also explores how evidence was tracked and surveillance integrated into the overall evidence workflow by using messaging apps and communication platforms. Finally, we consider the implications for living guidelines and reflect on factors that contributed to the feasibility of daily surveillance: the clearly defined scope of the guidelines; focusing efforts on maximum impact; minimizing duplication by partnering with others; setting up simple but effective processes for managing evidence; and the value of continuous involvement of personnel from the outset. Australia's living COVID‐19 guidelines were underpinned by a novel approach to evidence surveillance that was feasible and essential in maintaining COVID‐19 living guidelines for over 3 years.

show abstract

Section: Discussionmentioning

confidence: 99%

Designing, refining and reflecting on 3 years of daily evidence surveillance for Australia's living national COVID‐19 guidelines

McDonald,

Turner

2024

Cochrane Evidence Synthesis and Methods

View full text Add to dashboard Cite

show abstract

“…These have concentrated on evidence searching (Khalil et al, 2022) and selection given how demanding it is for humans to maintain truly upto-date evidence (Crequit et al, 2020;Thomas et al, 2021). Cochrane has deployed machine learning to identify randomized controlled trials (RCTs) and studies related to COVID-19 (Shemilt et al, 2022;Thomas et al, 2021), but such tools are not yet commonly used (Nguyen et al, 2022). The routine integration of automation tools in the development of future evidence syntheses should not displace the interpretive part of the process.…”

Section: Influences On the State Of Evidence Synthesismentioning

confidence: 99%

Guidance to best tools and practices for systematic reviews

Kolaski

Logan

Ioannidis

2023

British J Pharmacology

View full text Add to dashboard Cite

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well‐documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in‐depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

show abstract

“…These have concentrated on evidence searching 40 and selection given how demanding it is for humans to maintain truly up-to-date evidence 2,41 . Cochrane has deployed machine learning to identify randomized controlled trials (RCTs) 2 and studies related to COVID-19 42 , but such tools are not yet commonly used 43 . The routine integration of automation tools in the development of future evidence syntheses should not displace the interpretive part of the process.…”

Section: Influences On the State Of Evidence Synthesismentioning

confidence: 99%

Guidance to Best Tools and Practices for Systematic Reviews

2023

View full text Add to dashboard Cite

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

show abstract

Machine learning reduced workload for the Cochrane COVID-19 Study Register: development and evaluation of the Cochrane COVID-19 Study Classifier

Cited by 19 publications

References 9 publications

Designing, refining and reflecting on 3 years of daily evidence surveillance for Australia's living national COVID‐19 guidelines

Designing, refining and reflecting on 3 years of daily evidence surveillance for Australia's living national COVID‐19 guidelines

Guidance to best tools and practices for systematic reviews

Guidance to Best Tools and Practices for Systematic Reviews

Contact Info

Product

Resources

About