Pre-deployment assessment of an AI model to assist radiologists in chest X-ray detection and identification of lead-less implanted electronic devices for pre-MRI safety screening: realized implementation needs and proposed operational solutions

Abstract

Purpose: Chest X-ray (CXR) use in pre-MRI safety screening, such as for lead-less implanted electronic device (LLIED) recognition, is common. To assist CXR interpretation, we "predeployed" an artificial intelligence (AI) model to assess (1) accuracies in LLIED-type (and consequently safety-level) identification, (2) safety implications of LLIED nondetections or misidentifications, (3) infrastructural or workflow requirements, and (4) demands related to model adaptation to real-world conditions.Approach: A two-tier cascading methodology for LLIED detection/localization and identification on a frontal CXR was applied to evaluate the performance of the original nine-class AI model. With the unexpected early appearance of LLIED types during simulated real-world trialing, retraining of a newer 12-class version preceded retrialing. A zero footprint (ZF) graphical user interface (GUI)/viewer with DICOM-based output was developed for inference-result display and adjudication, supporting end-user engagement and model continuous learning and/or modernization.Results: During model testing or trialing using both the nine-class and 12-class models, robust detection/localization was consistently 100%, with mAP 0.99 from fivefold cross-validation. Safety-level categorization was high during both testing (AUC ≥ 0.98 and ≥0.99, respectively) and trialing (accuracy 98% and 97%, respectively). LLIED-type identifications by the two models during testing (1) were 98.9% and 99.5% overall correct and (2) consistently showed AUC ≥ 0.92 (1.00 for 8/9 and 9/12 LLIED-types, respectively). Pre-deployment trialing of both models demonstrated overall type-identification accuracies of 94.5% and 95%, respectively. Of the small number of misidentifications, none involved MRI-stringently conditional or MRIunsafe types of LLIEDs. Optimized ZF GUI/viewer operations led to greater user-friendliness for radiologist engagement.Conclusions: Our LLIED-related AI methodology supports (1) 100% detection sensitivity, (2) high identification (including MRI-safety) accuracy, and (3) future model deployment with facilitated inference-result display and adjudication for ongoing model adaptation to future realworld experiences.

AI Algorithm Deployment and Utilization: Strategies to Facilitate Clinical Workflow Implementation and Integration

“…In this project, we showed that by using a common standards-based architecture and workflow the ability to deploy algorithms, whether vendor-based (commercial products or prototypes) or locally developed [7], can carry significant advantages. This approach enables support of complex operational requirements and allows both types of solutions to coexist in the same environment.…”

“…Being able to capture standards-based user feedback enables database-driven algorithm development and enhancement, as well as easy and structured data capture. This is illustrated with a locally developed algorithm for leadless implanted electronic device algorithm for determination of MRI safety based on Chest X-Rays [7]. This algorithm is trained to recognize potentially MRI-unsafe devices using two cascading AI models (the results of the first model for detection based on a fast R-CNN then feed into the second model for identification based on a multi-class CNN).…”

“…• Unsupported Input: The received image was not suitable for processing, and consequently, no result was produced. The details of processing activity (Figure 5) and results statistics (Table 2) follow: In this project, we showed that by using a common standards-based architecture and workflow the ability to deploy algorithms, whether vendor-based (commercial products or prototypes) or locally developed [7], can carry significant advantages. This approach enables support of complex operational requirements and allows both types of solutions to coexist in the same environment.…”

“…Such was the case with AIRC Chest CT, which was successfully and easily integrated into the existing infrastructure based on the locally developed CAII Viewer [7] using existing DICOM Secondary Capture and DICOM SR TID1500 outputs, the last providing more interaction capabilities with the AI results [Figure 4]. Connecting different AIRC applications to the same infrastructure was a helpful step in assessing the robustness of the established connections.…”

“…Figure 9a: AIRC Chest X-ray results showing a subtle lung lesion as displayed in the locally developed CAII Viewer[7]. The CAII Viewer can display a wide range of DICOM formats, including non-editable Secondary Captures (as shown here), as well as DICOM SR outputs (left-hand-side menu) for more control by the end-user.…”