Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension

Rivera, Samantha Cruz; Liu, Xiaoxuan; Chan, An Wen; Denniston, Alastair K; Calvert, Melanie; Ashrafian, Hutan; Beam, Andrew L.; Collins, Gary S.; Darzi, Ara; Deeks, Jonathan J; ElZarrad, M. Khair; Espinoza, Cynthia; Esteva, Andre; Faes, Livia; Ruffano, Lavinia Ferrante di; Fletcher, John; Golub, Robert M.; Harvey, Hugh; Haug, Charlotte; Holmes, Chris; Jonas, Adrian; Keane, Pearse Andrew; Kelly, Christopher; Lee, A Y; Lee, C S; Manna, Elaine; Matcham, James; McCradden, Melissa; Moher, David; Monteiro, João Filipe G.; Mulrow, Cindy; Oakden-Rayner, Luke; Paltoo, Dina N.; Panico, Maria Beatrice; Price, Gary; Rowley, Samuel; Savage, Rock H.; Sarkar, Rupa; Vollmer, Sebastian J.; Yau, Christopher

doi:10.1016/s2589-7500(20)30219-3

Cited by 179 publications

(121 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Studies developing AI models should adhere to the TRIPOD reporting guidelines 22 , PROBAST 23 , or, ideally, recent AI-specific guidelines. These include the guidelines for transparency, reproducibility, ethics, and effectiveness (TREE) 42 , CONSORT-AI (Consolidated Standards of Reporting Trials-Artificial Intelligence) 43 , and SPIRIT-AI (Standard Protocol Items: Recommendations for Interventional Trials-Artificial Intelligence) 44 . While the above guidelines provide comprehensive explanations and elaborations, we emphasise hereinafter several common problematic areas within the reviewed studies and recommendations for future studies.…”

Section: Discussionmentioning

confidence: 99%

Artificial Intelligence Applications for COVID-19 in Intensive Care and Emergency Settings: A Systematic Review

Chee

Ong

Siddiqui

et al. 2021

Preprint

View full text Add to dashboard Cite

Background: Little is known about the role of artificial intelligence (AI) as a decisive technology in the clinical management of COVID-19 patients. We aimed to systematically review and critically appraise the current evidence on AI applications for COVID-19 in intensive care and emergency settings, focusing on methods, reporting standards, and clinical utility. Methods: We systematically searched PubMed, Embase, Scopus, CINAHL, IEEE Xplore, and ACM Digital Library databases from inception to 1 October 2020, without language restrictions. We included peer-reviewed original studies that applied AI for COVID-19 patients, healthcare workers, or health systems in intensive care, emergency or prehospital settings. We assessed predictive modelling studies using PROBAST (prediction model risk of bias assessment tool) and a modified TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) statement for AI. We critically appraised the methodology and key findings of all other studies. Results: Of fourteen eligible studies, eleven developed prognostic or diagnostic AI predictive models, all of which were assessed to be at high risk of bias. Common pitfalls included inadequate sample sizes, poor handling of missing data, failure to account for censored participants, and weak validation of models. Studies had low adherence to reporting guidelines, with particularly poor reporting on model calibration and blinding of outcome and predictor assessment. Of the remaining three studies, two evaluated the prognostic utility of deep learning-based lung segmentation software and one studied an AI-based system for resource optimisation in the ICU. These studies had similar issues in methodology, validation, and reporting. Conclusions: Current AI applications for COVID-19 are not ready for deployment in acute care settings, given their limited scope and poor quality. Our findings underscore the need for improvements to facilitate safe and effective clinical adoption of AI applications, for and beyond the COVID-19 pandemic.

show abstract

Section: Discussionmentioning

confidence: 99%

Artificial Intelligence Applications for COVID-19 in Intensive Care and Emergency Settings: A Systematic Review

Chee

Ong

Siddiqui

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…To address this, the SPIRIT-AI and CONSORT-AI Steering Group announced in October 2019 an initiative to develop evidencebased extensions for clinical trial protocols and reports involving AI interventions [19]. The SPIRIT-AI [20][21][22] and CONSORT-AI [23][24][25] guidelines have since been developed in accordance with the EQUATOR (Enhancing the Quality and Transparency of Health Research) Network recommendations. The guidelines were developed using a Delphi methodology with an international multidisciplinary consortium.…”

Section: The Spirit-ai and Consort-ai Guidelinesmentioning

confidence: 99%

“…The SPIRIT-AI [20][21][22] and CONSORT-AI [23][24][25] guidelines include 15 and 14 new items, respectively, that should be routinely reported in addition to the core SPIRIT 2013 [13] and CONSORT 2010 [14] items. For example, the new guidelines recommend that investigators should provide clear descriptions of the AI intervention, including instructions and skills required for use, the study setting in which the AI intervention is integrated, the handling of inputs and outputs of the AI intervention, the human-AI interaction, and the analysis of error cases.…”

Section: The Spirit-ai and Consort-ai Guidelinesmentioning

confidence: 99%

“…The SPIRIT-AI [20][21][22] and CONSORT-AI [23][24][25] guidelines have the potential to improve the quality of clinical trials of AI systems in health, through improvements in design and delivery, and the completeness and transparency of their reporting. It is, however, important to appreciate the context in which these guidelines sit.…”

Section: The Spirit-ai and Consort-ai Guidelinesmentioning

confidence: 99%

“…It is, however, important to appreciate the context in which these guidelines sit. First, the new items within SPIRIT-AI [20][21][22] and CONSORT-AI [23][24][25] are all extensions or elaborations rather than substitutes for the core items. Core considerations addressed by SPIRIT 2013 [13] and CONSORT 2010 [14] remain important in all clinical trial protocols and reports, regardless of the intervention itself.…”

Section: The Spirit-ai and Consort-ai Guidelinesmentioning

confidence: 99%

See 2 more Smart Citations

Reporting guidelines for clinical trials of artificial intelligence interventions: the SPIRIT-AI and CONSORT-AI guidelines

Ibrahim

Liu

Rivera

et al. 2021

Trials

Self Cite

View full text Add to dashboard Cite

Background The application of artificial intelligence (AI) in healthcare is an area of immense interest. The high profile of ‘AI in health’ means that there are unusually strong drivers to accelerate the introduction and implementation of innovative AI interventions, which may not be supported by the available evidence, and for which the usual systems of appraisal may not yet be sufficient. Main text We are beginning to see the emergence of randomised clinical trials evaluating AI interventions in real-world settings. It is imperative that these studies are conducted and reported to the highest standards to enable effective evaluation because they will potentially be a key part of the evidence that is used when deciding whether an AI intervention is sufficiently safe and effective to be approved and commissioned. Minimum reporting guidelines for clinical trial protocols and reports have been instrumental in improving the quality of clinical trials and promoting completeness and transparency of reporting for the evaluation of new health interventions. The current guidelines—SPIRIT and CONSORT—are suited to traditional health interventions but research has revealed that they do not adequately address potential sources of bias specific to AI systems. Examples of elements that require specific reporting include algorithm version and the procedure for acquiring input data. In response, the SPIRIT-AI and CONSORT-AI guidelines were developed by a multidisciplinary group of international experts using a consensus building methodological process. The extensions include a number of new items that should be reported in addition to the core items. Each item, where possible, was informed by challenges identified in existing studies of AI systems in health settings. Conclusion The SPIRIT-AI and CONSORT-AI guidelines provide the first international standards for clinical trials of AI systems. The guidelines are designed to ensure complete and transparent reporting of clinical trial protocols and reports involving AI interventions and have the potential to improve the quality of these clinical trials through improvements in their design and delivery. Their use will help to efficiently identify the safest and most effective AI interventions and commission them with confidence for the benefit of patients and the public.

show abstract

Checklist for Evaluation of Image-Based Artificial Intelligence Reports in Dermatology

et al. 2022

View full text Add to dashboard Cite

show abstract

Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension

Cited by 179 publications

References 48 publications

Artificial Intelligence Applications for COVID-19 in Intensive Care and Emergency Settings: A Systematic Review

Artificial Intelligence Applications for COVID-19 in Intensive Care and Emergency Settings: A Systematic Review

Reporting guidelines for clinical trials of artificial intelligence interventions: the SPIRIT-AI and CONSORT-AI guidelines

Checklist for Evaluation of Image-Based Artificial Intelligence Reports in Dermatology

Contact Info

Product

Resources

About