Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study

Soboczenski, Frank; Trikalinos, Thomas A; Kuiper, Joël; Bias, Randolph G.; Wallace, Byron C.; Marshall, Iain J.

doi:10.1186/s12911-019-0814-z

Cited by 35 publications

(33 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Recently, machine learning (ML) algorithms have used computational methods to “learn” information directly from data, and their performance has been shown to improve proportionally with the number of high-quality samples [ 8 ]. ML algorithms have been applied in different aspects of medicine [ 9 , 10 ], including earlier disease detection [ 11 , 12 ], improve diagnosis accuracy [ 13 – 16 ], identification of new physiological observations or patterns [ 17 ], development of personalized diagnostics and/or therapeutic approaches [ 18 , 19 ], research purposes [ 20 ], etc.…”

Section: Introductionmentioning

confidence: 99%

Machine learning versus conventional clinical methods in guiding management of heart failure patients—a systematic review

et al. 2020

View full text Add to dashboard Cite

Machine learning (ML) algorithms “learn” information directly from data, and their performance improves proportionally with the number of high-quality samples. The aim of our systematic review is to present the state of the art regarding the implementation of ML techniques in the management of heart failure (HF) patients. We manually searched MEDLINE and Cochrane databases as well the reference lists of the relevant review studies and included studies. Our search retrieved 122 relevant studies. These studies mainly refer to (a) the role of ML in the classification of HF patients into distinct categories which may require a different treatment strategy, (b) discrimination of HF patients from the healthy population or other diseases, (c) prediction of HF outcomes, (d) identification of HF patients from electronic records and identification of HF patients with similar characteristics who may benefit form a similar treatment strategy, (e) supporting the extraction of important data from clinical notes, and (f) prediction of outcomes in HF populations with implantable devices (left ventricular assist device, cardiac resynchronization therapy). We concluded that ML techniques may play an important role for the efficient construction of methodologies for diagnosis, management, and prediction of outcomes in HF patients. Electronic supplementary material The online version of this article (10.1007/s10741-020-10007-3) contains supplementary material, which is available to authorized users.

show abstract

Section: Introductionmentioning

confidence: 99%

Machine learning versus conventional clinical methods in guiding management of heart failure patients—a systematic review

et al. 2020

View full text Add to dashboard Cite

show abstract

“…However, a recent study found that using RobotReviewer as a semi‐autonomous tool was nonetheless quicker on average than no automation by 69 seconds (for all studies analyzed). This 2019 study, outlined in BMC Medical Informatics and Decision Making , was performed with a small sample size and used a sample of reviewers not typical of a team of systematic reviewers (5 of the 41 volunteers had no experience with systematic reviews and nine had no experience with the risk of bias tool).…”

Section: Discussionmentioning

confidence: 91%

Comparing machine and human reviewers to evaluate the risk of bias in randomized controlled trials

Armijo‐Olivo

Craig

Campbell

2020

Research Synthesis Methods

View full text Add to dashboard Cite

Background: Evidence from new health technologies is growing, along with demands for evidence to inform policy decisions, creating challenges in completing health technology assessments (HTAs)/systematic reviews (SRs) in a timely manner. Software can decrease the time and burden by automating the process, but evidence validating such software is limited. We tested the accuracy of RobotReviewer, a semi-autonomous risk of bias (RoB) assessment tool, and its agreement with human reviewers.Methods: Two reviewers independently conducted RoB assessments on a sample of randomized controlled trials (RCTs), and their consensus ratings were compared with those generated by RobotReviewer. Agreement with the human reviewers was assessed using percent agreement and weighted kappa (κ). The accuracy of RobotReviewer was also assessed by calculating the sensitivity, specificity, and area under the curve in comparison to the consensus agreement of the human reviewers. Results:The study included 372 RCTs. Inter-rater reliability ranged from κ = −0.06 (no agreement) for blinding of participants and personnel to κ = 0.62 (good agreement) for random sequence generation (excluding overall RoB).RobotReviewer was found to use a high percentage of "irrelevant supporting quotations" to complement RoB assessments for blinding of participants and personnel (72.6%), blinding of outcome assessment (70.4%), and allocation concealment (54.3%). Conclusion:RobotReviewer can help with risk of bias assessment of RCTs but cannot replace human evaluations. Thus, reviewers should check and validate RoB assessments from RobotReviewer by consulting the original article when not relevant supporting quotations are provided by RobotReviewer. This consultation is in line with the recommendation provided by the developers. K E Y W O R D Sartificial intelligence, health technology assessment (HTA), inter-rater reliability, randomized controlled trial, risk of bias, systematic review | INTRODUCTIONKnowledge synthesis products such as health technology assessments (HTAs) and systematic reviews (SRs) are important sources of high-quality information for policy makers, and the production of these assessments and reports requires expertise, human resources, and time. 1 Given the proliferation of health technologies and

show abstract

“…For example, reviewers might judge the tool’s accuracy as sufficient or helpful for its supportive use when facing a high number of studies for risk of bias assessment or facing studies that are reported in a different structure than in scientific journals such as grey literature publications. Furthermore, a recent study indicates that using RobotReviewer might be less time consuming than manual assessment by human reviewers (Soboczenski et al, 2019).…”

Section: Discussionmentioning

confidence: 99%

“…For example, reviewers might judge the tool's accuracy as sufficient or helpful for its supportive use when facing a high number of studies for risk of bias assessment or facing studies that are reported in a different structure than in scientific journals such as grey literature publications. Furthermore, a recent study indicates that using RobotReviewer might be less time consuming than manual assessment by human reviewers (Soboczenski et al, 2019). Third, although the tool has not been developed to replace assessment by human reviewers (Marshall et al, 2016), research shows that RobotReviewer's reliability is comparable to that between two independent human reviewers (Hartling et al, 2013).…”

Section: Discussionmentioning

confidence: 99%

Agreement in Risk of Bias Assessment Between RobotReviewer and Human Reviewers: An Evaluation Study on Randomised Controlled Trials in Nursing‐Related Cochrane Reviews

Hirt

Meichlinger

Schumacher

et al. 2021

J of Nursing Scholarship

View full text Add to dashboard Cite

Purpose RobotReviewer is a machine learning system for semi‐automated assistance in risk of bias assessment. The tools’s performance in randomized controlled trials (RCTs) in the field of nursing remains unknown. We aimed therefore to evaluate the agreement in risk of bias assessment between RobotReviewer and human reviewers. Design Evaluation study using a retrospective diagnostic design. Methods We used RobotReviewer as the index test and human reviewers’ risk of bias assessment reported in Cochrane reviews as the reference test. A convenience sample of electronically available English‐language full texts of RCTs included in Cochrane reviews with nurs* in the title were eligible for inclusion. In this context, we assessed random sequence generation, allocation concealment, and blinding (personnel or participants and assessors) corresponding to Cochrane risk of bias version 2011. Two independent research teams performed and double‐checked data extraction and analysis. We calculated sensitivity, specificity, receiver operating characteristic (ROC) curve, the area under the ROC curve, predictive values, observed percentage of agreement, and Cohen’s kappa (including confidence intervals, if applicable). Findings The selection process yielded 190 RCTs published between 1958 and 2016 in 23 Cochrane reviews published between 2000 and 2018. Missing assessments of risk of bias domains in Cochrane reviews or RobotReviewer yielded varying sample sizes per risk of bias domain. Sensitivity ranged from 0.44 to 0.88 and specificity from 0.48 to 0.95. Positive predictive value was highest for allocation concealment (0.79) and lowest for blinding assessors (0.25). Cohen’s kappa was moderate for randomization (0.52), allocation concealment (0.60), and for blinding of personnel/patients (0.43). Blinding of outcome assessors had only slight agreement (0.04). Conclusions This is the first evaluation of risk of bias assessment by RobotReviewer in RCTs included in nursing‐related Cochrane reviews. It yielded a moderate degree of agreement with human reviewers for randomization and allocation concealment, and an adequate sensitivity for detecting low risk of selection bias. Clinical Relevance Based on our results, using the RobotReviewer for risk of bias assessment in RCTs can be supportive in some risk of bias domains. However, human reviewers should supervise the semi‐automated assessment process.

show abstract

Machine learning to help researchers evaluate biases in clinical trials: a prospective, randomized user study

Cited by 35 publications

References 18 publications

Machine learning versus conventional clinical methods in guiding management of heart failure patients—a systematic review

Machine learning versus conventional clinical methods in guiding management of heart failure patients—a systematic review

Comparing machine and human reviewers to evaluate the risk of bias in randomized controlled trials

Agreement in Risk of Bias Assessment Between RobotReviewer and Human Reviewers: An Evaluation Study on Randomised Controlled Trials in Nursing‐Related Cochrane Reviews

Contact Info

Product

Resources

About