Multi-Institution Encrypted Medical Imaging Ai Validation Without Data Sharing

Soin, Arjun; Takhar, Rohit; Chandran, Nishanth; Gupta, Divya; Alvarez-Valle, Javier; Sharma, Rahul; Mahajan, Vidur; Lungren, Matthew P.

doi:10.2139/ssrn.3973993

Cited by 6 publications

(11 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Of note, whereas five bronchial NGTs were incorrectly assessed by unaided junior physicians as safe for feeding, there were no incorrect feeding decisions with AI support. Overall, our results suggest that improvements in safety and performance can be achieved through synergistic decision support in fast-paced clinical environments ( 14 ).…”

Section: Discussionmentioning

confidence: 72%

“…Of particular interest was assessment of how model performance changes over the model’s life cycle. Although traditional performance drift approaches involve monitoring a metric (eg, AUC), this is not practical in a health care setting ( 14 ). We introduced a simple real-time drift detector that uses low-dimensional embeddings of the latent model space and calculates an interpretable drift P value using the two-sample Kolmogorov-Smirnov test at an image level.…”

Section: Discussionmentioning

confidence: 99%

“…Our method enables real-time monitoring of model performance in the absence of contemporaneous ground truths. It is conceptually similar to the method used by Soin et al ( 14 ); however, it does not use a variational autoencoder to detect changes in input data, and instead uses a twin neural network (Ivis) to represent latent model information. As such, it can be easily adapted to nonimaging data.…”

Section: Discussionmentioning

confidence: 99%

“…In conclusion, the developed deep learning tool for detection of NGT misplacement on chest radiographs may aid junior physicians in clinical decision-making regarding safety of feeding for patients with NGTs. Future work will include formal development and benchmarking of our drift detection system with tools such as CheXStray ( 14 ). In addition, we plan to introduce an NGT segmentation component to the ensemble.…”

Section: Discussionmentioning

confidence: 99%

“…They present attractive clinical decision support tools, contributing to improved diagnostic accuracies among emergency medicine clinicians ( 13 ) and radiologists ( 12 ). Despite their general use, artificial neural network performance in dynamic real-world systems can be disrupted because of concept drift, caused by extrinsic factors such as changes in imaging workflows, instrument calibration, or software updates ( 14 ). Furthermore, application of artificial neural networks to detection of NGT malposition has been limited to detection of bronchial placements ( 15 ) and NGT segmentation ( 16 ).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

An Artificial Neural Network for Nasogastric Tube Position Decision Support

Drozdov

Dixon

Szubert

et al. 2023

Radiology: Artificial Intelligence

View full text Add to dashboard Cite

Purpose To develop and validate a deep learning model for detection of nasogastric tube (NGT) malposition on chest radiographs and assess model impact as a clinical decision support tool for junior physicians to help determine whether feeding can be safely performed in patients (feed/do not feed). Materials and Methods A neural network ensemble was pretrained on 1 132 142 retrospectively collected (June 2007–August 2019) frontal chest radiographs and further fine-tuned on 7081 chest radiographs labeled by three radiologists. Clinical relevance was assessed on an independent set of 335 images. Five junior emergency medicine physicians assessed chest radiographs and made feed/do not feed decisions without and with artificial intelligence (AI)-generated NGT malposition probabilities placed above chest radiographs. Decisions from the radiologists served as ground truths. Model performance was evaluated using receiver operating characteristic analysis. Agreement between junior physician and radiologist decision was determined using the Cohen κ coefficient. Results In the testing set, the ensemble achieved area under the receiver operating characteristic curve values of 0.82 (95% CI: 0.78, 0.86), 0.77 (95% CI: 0.71, 0.83), and 0.98 (95% CI: 0.96, 1.00) for satisfactory, malpositioned, and bronchial positions, respectively. In the clinical evaluation set, mean interreader agreement for feed/do not feed decisions among junior physicians was 0.65 ± 0.03 (SD) and 0.77 ± 0.13 without and with AI support, respectively. Mean agreement between junior physicians and radiologists was 0.53 ± 0.05 (unaided) and 0.65 ± 0.09 (AI-aided). Conclusion A simple classifier for NGT malposition may help junior physicians determine the safety of feeding in patients with NGTs. Keywords: Neural Networks, Feature Detection, Supervised Learning, Machine Learning Supplemental material is available for this article. Published under a CC BY 4.0 license.

show abstract

Section: Discussionmentioning

confidence: 72%

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

An Artificial Neural Network for Nasogastric Tube Position Decision Support

Drozdov

Dixon

Szubert

et al. 2023

Radiology: Artificial Intelligence

View full text Add to dashboard Cite

show abstract

A quality assurance framework for routine monitoring of deep learning cardiac substructure computed tomography segmentation models in radiotherapy

Jin,

Hao,

Hilliard

et al. 2023

Medical Physics

View full text Add to dashboard Cite

BackgroundFor autosegmentation models, the data used to train the model (e.g., public datasets and/or vendor‐collected data) and the data on which the model is deployed in the clinic are typically not the same, potentially impacting the performance of these models by a process called domain shift. Tools to routinely monitor and predict segmentation performance are needed for quality assurance. Here, we develop an approach to perform such monitoring and performance prediction for cardiac substructure segmentation.PurposeTo develop a quality assurance (QA) framework for routine or continuous monitoring of domain shift and the performance of cardiac substructure autosegmentation algorithms.MethodsA benchmark dataset consisting of computed tomography (CT) images along with manual cardiac substructure delineations of 241 breast cancer radiotherapy patients were collected, including one “normal” image domain of clean images and five “abnormal” domains containing images with artifact (metal, contrast), pathology, or quality variations due to scanner protocol differences (field of view, noise, reconstruction kernel, and slice thickness). The QA framework consisted of an image domain shift detector which operated on the input CT images and a shape quality detector on the output of an autosegmentation model, and a regression model for predicting autosegmentation model performance. The image domain shift detector was composed of a trained denoising autoencoder (DAE) and two hand‐engineered image quality features to detect normal versus abnormal domains in the input CT images. The shape quality detector was a variational autoencoder (VAE) trained to estimate the shape quality of the auto‐segmentation results. The output from the image domain shift and shape quality detectors was used to train a regression model to predict the per‐patient segmentation accuracy, measured by Dice coefficient similarity (DSC) to physician contours. Different regression techniques were investigated including linear regression, Bagging, Gaussian process regression, random forest, and gradient boost regression. Of the 241 patients, 60 were used to train the autosegmentation models, 120 for training the QA framework, and the remaining 61 for testing the QA framework. A total of 19 autosegmentation models were used to evaluate QA framework performance, including 18 convolutional neural network (CNN)‐based and one transformer‐based model.ResultsWhen tested on the benchmark dataset, all abnormal domains resulted in a significant DSC decrease relative to the normal domain for CNN models (), but only for some domains for the transformer model. No significant relationship was found between the performance of an autosegmentation model and scanner protocol parameters () except noise (). CNN‐based autosegmentation models demonstrated a decreased DSC ranging from 0.07 to 0.41 with added noise, while the transformer‐based model was not significantly affected (ANOVA, ). For the QA framework, linear regression models with bootstrap aggregation resulted in the highest mean absolute error (MAE) of 0.041 ± 0.002, in predicted DSC (relative to true DSC between autosegmentation and physician). MAE was lowest when combining both input (image) detectors and output (shape) detectors compared to output detectors alone.ConclusionsA QA framework was able to predict cardiac substructure autosegmentation model performance for clinically anticipated “abnormal” domain shifts.

show abstract

Developing, purchasing, implementing and monitoring AI tools in radiology: Practical considerations. A multi‐society statement from the ACR, CAR, ESR, RANZCR & RSNA

Brady,

Allen,

Chong

et al. 2024

J Med Imag Rad Onc

View full text Add to dashboard Cite

SummaryArtificial Intelligence (AI) carries the potential for unprecedented disruption in radiology, with possible positive and negative consequences. The integration of AI in radiology holds the potential to revolutionize healthcare practices by advancing diagnosis, quantification, and management of multiple medical conditions. Nevertheless, the ever‐growing availability of AI tools in radiology highlights an increasing need to critically evaluate claims for its utility and to differentiate safe product offerings from potentially harmful, or fundamentally unhelpful ones. This multi‐society paper, presenting the views of Radiology Societies in the USA, Canada, Europe, Australia, and New Zealand, defines the potential practical problems and ethical issues surrounding the incorporation of AI into radiological practice. In addition to delineating the main points of concern that developers, regulators, and purchasers of AI tools should consider prior to their introduction into clinical practice, this statement also suggests methods to monitor their stability and safety in clinical use, and their suitability for possible autonomous function. This statement is intended to serve as a useful summary of the practical issues which should be considered by all parties involved in the development of radiology AI resources, and their implementation as clinical tools.

show abstract

Multi-Institution Encrypted Medical Imaging Ai Validation Without Data Sharing

Cited by 6 publications

References 12 publications

An Artificial Neural Network for Nasogastric Tube Position Decision Support

An Artificial Neural Network for Nasogastric Tube Position Decision Support

A quality assurance framework for routine monitoring of deep learning cardiac substructure computed tomography segmentation models in radiotherapy

Developing, purchasing, implementing and monitoring AI tools in radiology: Practical considerations. A multi‐society statement from the ACR, CAR, ESR, RANZCR & RSNA

Contact Info

Product

Resources

About