Randomized Clinical Trials of Machine Learning Interventions in Health Care

Abstract

ImportanceDespite the potential of machine learning to improve multiple aspects of patient care, barriers to clinical adoption remain. Randomized clinical trials (RCTs) are often a prerequisite to large-scale clinical adoption of an intervention, and important questions remain regarding how machine learning interventions are being incorporated into clinical trials in health care.ObjectiveTo systematically examine the design, reporting standards, risk of bias, and inclusivity of RCTs for medical machine learning interventions.Evidence ReviewIn this systematic review, the Cochrane Library, Google Scholar, Ovid Embase, Ovid MEDLINE, PubMed, Scopus, and Web of Science Core Collection online databases were searched and citation chasing was done to find relevant articles published from the inception of each database to October 15, 2021. Search terms for machine learning, clinical decision-making, and RCTs were used. Exclusion criteria included implementation of a non-RCT design, absence of original data, and evaluation of nonclinical interventions. Data were extracted from published articles. Trial characteristics, including primary intervention, demographics, adherence to the CONSORT-AI reporting guideline, and Cochrane risk of bias were analyzed.FindingsLiterature search yielded 19 737 articles, of which 41 RCTs involved a median of 294 participants (range, 17-2488 participants). A total of 16 RCTS (39%) were published in 2021, 21 (51%) were conducted at single sites, and 15 (37%) involved endoscopy. No trials adhered to all CONSORT-AI standards. Common reasons for nonadherence were not assessing poor-quality or unavailable input data (38 trials [93%]), not analyzing performance errors (38 [93%]), and not including a statement regarding code or algorithm availability (37 [90%]). Overall risk of bias was high in 7 trials (17%). Of 11 trials (27%) that reported race and ethnicity data, the median proportion of participants from underrepresented minority groups was 21% (range, 0%-51%).Conclusions and RelevanceThis systematic review found that despite the large number of medical machine learning–based algorithms in development, few RCTs for these technologies have been conducted. Among published RCTs, there was high variability in adherence to reporting standards and risk of bias and a lack of participants from underrepresented minority groups. These findings merit attention and should be considered in future RCT design and reporting.

Performance of ChatGPT on USMLE: Potential for AI-Assisted Medical Education Using Large Language Models

“…As AI becomes increasingly proficient, it will soon become ubiquitous, transforming clinical medicine across all healthcare sectors. Investigation of AI has now entered into the era of randomized controlled trials 15 . Additionally, a profusion of pragmatic and observational studies supports a versatile role of AI in virtually all medical disciplines and specialities by improving risk assessment 16,17 , data reduction, clinical decision support 18,19 , operational efficiency, and patient communication 20,21 .…”

Must-have Qualities of Clinical Research on Artificial Intelligence and Machine Learning

“…Presenting experimental details and results with sufficient thoroughness remains an issue in AI research within the medical field. 95 This limitation is relevant as it hinders the build-up of trust in physicians and ultimately patients, limiting clinical adoption of tools based on ML technologies. Accordingly, several entities, including scientific societies, journal editorial offices, and domain experts, have attempted to set common reporting standards for AI studies 61 , 96 , 97 , 98 , 99 These have taken the form of white/position papers or checklists, the second of which may include a quantitative methodological quality assessment, as in the case of the Radiomics Quality Score.…”

Technical Skill Assessment in Minimally Invasive Surgery Using Artificial Intelligence: A Systematic Review

“…Based on kinematic data performance metrics of technical skills were developed 8 . AI is a very promising technology that is widely adopted in medicine 9,10 . For example, AI is able to detect diabetic retinopathy 11,12 and to screen for lung cancer 13 and malignant skin cancer 14 with an accuracy comparable to expert clinician screening.…”