Inclusion of Clinicians in the Development and Evaluation of Clinical Artificial Intelligence Tools: A Systematic Literature Review

Abstract

The application of machine learning (ML) and artificial intelligence (AI) in healthcare domains has received much attention in recent years, yet significant questions remain about how these new tools integrate into frontline user workflow, and how their design will impact implementation. Lack of acceptance among clinicians is a major barrier to the translation of healthcare innovations into clinical practice. In this systematic review, we examine when and how clinicians are consulted about their needs and desires for clinical AI tools. Forty-five articles met criteria for inclusion, of which 24 were considered design studies. The design studies used a variety of methods to solicit and gather user feedback, with interviews, surveys, and user evaluations. Our findings show that tool designers consult clinicians at various but inconsistent points during the design process, and most typically at later stages in the design cycle (82%, 19/24 design studies). We also observed a smaller amount of studies adopting a human-centered approach and where clinician input was solicited throughout the design process (22%, 5/24). A third (15/45) of all studies reported on clinician trust in clinical AI algorithms and tools. The surveyed articles did not universally report validation against the “gold standard” of clinical expertise or provide detailed descriptions of the algorithms or computational methods used in their work. To realize the full potential of AI tools within healthcare settings, our review suggests there are opportunities to more thoroughly integrate frontline users’ needs and feedback in the design process.

Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology

“…Aiming to narrow the gap between (multimodal) frontier AI model advances and their successful translation into clinical practice [32,44,97,100,130,132,149], our work engaged in early phase design and user research to identify and co-create clinically relevant use cases of VLM capabilities for radiology. Below, we first discuss the findings from our user feedback sessions, detailing design requirements for each use case.…”

“…Despite great AI advances both in natural language processing and image-based analysis, translating recent research and development successes into clinical practice remains challenging [32,44,97,100,121,130,132,145,149,150]. Factors hindering successful AI implementation in radiology are wide ranging and include: skepticism due to inconsistent AI performance; lack of trust and overreliance in AI-generated outputs; and the need for clinical effectiveness trials (cf.…”

Electronic health record-based prediction models for in-hospital adverse drug event diagnosis or prognosis: a systematic review

“…Medical experts are indispensable in this process. 25 , 57 , 58 Involving them, along with prediction model experts and implementation experts, serves to identify clinically relevant problems that may be addressed using implementable valid prediction models. Reporting on the process and the involved key stakeholders is important as it increases the support across the different disciplines.…”

Sketching AI Concepts with Capabilities and Examples: AI Innovation in the Intensive Care Unit

“…In recent years, the advances in AI and the availability of high-density datasets, such as patient electronic health records (EHR), have enabled a new wave of innovations, spanning systems that support diagnosis, treatment recommendations, and automated documentation. However, similar to other domains, AI systems in healthcare have a poor track record; they largely fail when moving from research labs to clinical practice [25,32,61,107,127,137,141]. The clinical utility of these systems remain often unclear [42,44,120]; as a result, clinicians often do not use them [119,137].…”