Developing specific reporting guidelines for diagnostic accuracy studies assessing AI interventions: The STARD-AI Steering Group

Sounderajah, Viknesh; Ashrafian, Hutan; Aggarwal, Ravi; Fauw, Jeffrey De; Denniston, Alastair K; Greaves, Felix; Karthikesalingam, Alan; King, Dominic; Liu, Xiaoxuan; Markar, Sheraz; McInnes, Matthew D. F.; Panch, Trishan; Pearson‐Stuttard, Jonathan; Ting, Daniel; Golub, Robert M.; Moher, David; Bossuyt, Patrick M.; Darzi, Ara

doi:10.1038/s41591-020-0941-1

Cited by 172 publications

(131 citation statements)

References 5 publications

Supporting

Mentioning

126

Contrasting

Unclassified

Order By: Relevance

“…This extension is particularly aimed at investigators and readers reporting or appraising clinical trials; however, it may also serve as useful guidance for developers of AI interventions in earlier validation stages of an AI system. Investigators seeking to report studies developing and validating the diagnostic and predictive properties of AI models should refer to TRIPOD-ML (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis - Machine Learning) and STARD-AI (Standards for Reporting Diagnostic accuracy studies - Artificial Intelligence), both of which are currently under development 3262. Other potentially relevant guidelines are registered with the EQUATOR network, which are agnostic to study design 63.…”

Section: Discussionmentioning

confidence: 99%

Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI Extension

Liu

Rivera

Moher

et al. 2020

BMJ

Self Cite

231

187

View full text Add to dashboard Cite

The CONSORT 2010 (Consolidated Standards of Reporting Trials) statement provides minimum guidelines for reporting randomised trials. Its widespread use has been instrumental in ensuring transparency when evaluating new interventions. More recently, there has been a growing recognition that interventions involving artificial intelligence (AI) need to undergo rigorous, prospective evaluation to demonstrate impact on health outcomes.The CONSORT-AI extension is a new reporting guideline for clinical trials evaluating interventions with an AI component. It was developed in parallel with its companion statement for clinical trial protocols: SPIRIT-AI. Both guidelines were developed through a staged consensus process, involving a literature review and expert consultation to generate 29 candidate items, which were assessed by an international multi-stakeholder group in a two-stage Delphi survey (103 stakeholders), agreed on in a two-day consensus meeting (31 stakeholders) and refined through a checklist pilot (34 participants).The CONSORT-AI extension includes 14 new items, which were considered sufficiently important for AI interventions, that they should be routinely reported in addition to the core CONSORT 2010 items. CONSORT-AI recommends that investigators provide clear descriptions of the AI intervention, including instructions and skills required for use, the setting in which the AI intervention is integrated, the handling of inputs and outputs of the AI intervention, the human-AI interaction and providing analysis of error cases.CONSORT-AI will help promote transparency and completeness in reporting clinical trials for AI interventions. It will assist editors and peer-reviewers, as well as the general readership, to understand, interpret and critically appraise the quality of clinical trial design and risk of bias in the reported outcomes.

show abstract

Section: Discussionmentioning

confidence: 99%

Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI Extension

Liu

Rivera

Moher

et al. 2020

BMJ

Self Cite

231

187

View full text Add to dashboard Cite

show abstract

“…This extension is particularly aimed at investigators planning or conducting clinical trials; however, it may also serve as useful guidance for developers of AI interventions in earlier validation stages of an AI system. Investigators seeking to report studies developing and validating the diagnostic and predictive properties of AI models should refer to TRIPOD-ML (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis - Machine Learning)24 and STARD-AI (Standards for Reporting Diagnostic accuracy studies - Artificial Intelligence),51 both of which are currently under development. Other potentially relevant guidelines are registered with the EQUATOR network which are agnostic to study design 52.…”

Section: Discussionmentioning

confidence: 99%

Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI Extension

Rivera

Liu

Chan

et al. 2020

BMJ

Self Cite

210

194

View full text Add to dashboard Cite

The SPIRIT 2013 (The Standard Protocol Items: Recommendations for Interventional Trials) statement aims to improve the completeness of clinical trial protocol reporting, by providing evidence-based recommendations for the minimum set of items to be addressed. This guidance has been instrumental in promoting transparent evaluation of new interventions. More recently, there is a growing recognition that interventions involving artificial intelligence need to undergo rigorous, prospective evaluation to demonstrate their impact on health outcomes.The SPIRIT-AI extension is a new reporting guideline for clinical trials protocols evaluating interventions with an AI component. It was developed in parallel with its companion statement for trial reports: CONSORT-AI. Both guidelines were developed using a staged consensus process, involving a literature review and expert consultation to generate 26 candidate items, which were consulted on by an international multi-stakeholder group in a 2-stage Delphi survey (103 stakeholders), agreed on in a consensus meeting (31 stakeholders) and refined through a checklist pilot (34 participants).The SPIRIT-AI extension includes 15 new items, which were considered sufficiently important for clinical trial protocols of AI interventions. These new items should be routinely reported in addition to the core SPIRIT 2013 items. SPIRIT-AI recommends that investigators provide clear descriptions of the AI intervention, including instructions and skills required for use, the setting in which the AI intervention will be integrated, considerations around the handling of input and output data, the human-AI interaction and analysis of error cases.SPIRIT-AI will help promote transparency and completeness for clinical trial protocols for AI interventions. Its use will assist editors and peer-reviewers, as well as the general readership, to understand, interpret and critically appraise the design and risk of bias for a planned clinical trial.

show abstract

“…6 National Institute of Health Research BRC for Ophthalmology, Moorfields Eye Hospital NHS Foundation Trust, London, UK. 7 Centre for Regulatory Science and Innovation, Birmingham Health Partners, Birmingham, UK.…”

Section: Discussionmentioning

confidence: 99%

“…Future work is already underway to improve the standard of design and reporting for non-randomized studies including retrospective observational analysis and the development of prognostic models that depend upon AI. This work will soon lead to EQUATOR-supported guidelines specifically for both diagnostic test accuracy studies (STARD-AI [7]) and prognostic model evaluations (TRIPOD-ML [8]).…”

Section: Flattening the Hype Curve In Aimentioning

confidence: 99%

Going on up to the SPIRIT in AI: will new reporting guidelines for clinical trials of AI interventions improve their rigour?

Wicks

Liu

Denniston

2020

BMC Med

Self Cite

View full text Add to dashboard Cite

Developing specific reporting guidelines for diagnostic accuracy studies assessing AI interventions: The STARD-AI Steering Group

Cited by 172 publications

References 5 publications

Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI Extension

Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI Extension

Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI Extension

Going on up to the SPIRIT in AI: will new reporting guidelines for clinical trials of AI interventions improve their rigour?

Contact Info

Product

Resources

About