This article will undergo copyediting, layout, and proof review before it is published in its final version. Please note that during production of the final copyedited article, errors may be discovered which could affect the content.

Integration of artificial intelligence (AI) applications within clinical workflows is an important step for leveraging developed AI algorithms. In this report, generalizable components for deploying AI systems into clinical practice are described that were implemented in a clinical pilot study using lymphoscintigraphy examinations as a prospective use case (July 1, 2019-October 31, 2020). Deployment of the AI algorithm consisted of seven software components, as follows: (a) image delivery, (b) quality control, (c) a results database, (d) results processing, (e) results presentation and delivery, (f) error correction, and (g) a dashboard for performance monitoring. A total of 14 users used the system (faculty radiologists and trainees) to assess the degree of satisfaction with the components and overall workflow. Analyses included the assessment of the number of examinations processed, error rates, and corrections. The AI system processed 1748 lymphoscintigraphy examinations. The system enabled radiologists to correct 146 AI results, generating real-time corrections to the radiology report. All AI results and corrections were successfully stored in a database for downstream use by the various integration components. A dashboard allowed monitoring of the AI system performance in real time. All 14 survey respondents "somewhat agreed" or "strongly agreed" that the AI system was well integrated into the clinical workflow. In all, a framework of processes and components for integrating AI algorithms into clinical workflows was developed. The implementation described could be helpful for assessing and monitoring AI performance in clinical practice.

No abstract

Purpose Starting from a broad-based needs assessment and utilizing an image analysis algorithm (IAA) developed at our institution, the purpose of this study was to define generalizable building blocks necessary for the integration of any IAA into a clinical practice. Methods An IAA was developed in our institution to process lymphoscintigraphy exams. A team of radiologists defined a set of building blocks for integration of this IAA into clinical workflow. The building blocks served the following roles: (1) Timely delivery of images to the IAA, (2) quality control, (3) IAA results processing, (4) results presentation & delivery, (5) IAA error correction, (6) system performance monitoring, and (7) active learning. Utilizing these modules, the lymphoscintigraphy IAA was integrated into the clinical workflow at our institution. System performance was tested over a 1 month period, including assessment of number of exams processed and delivered, and error rates and corrections. Results From June 26-July 27, 2019, the building blocks were used to integrate IAA results from 132 lymphoscintigraphy exams into the clinical workflow, representing 100% of the exams performed during the time period. The system enabled radiologists to correct 21 of the IAA results. All results and corrections were successfully stored in a database. A dashboard allowed the development team to monitor system performance in real-time. Conclusions We describe seven building blocks that optimize the integration of IAAs into clinical workflow. The implementation of these building blocks in this study can be used to inform development of more robust, standards-based solutions.