Developing an Artificial Intelligence Model for Reading Chest X-rays: Protocol for a Prospective Validation Study

Catalina, Queralt Miró; Fuster-Casanovas, Aína; Solé-Casals, Jordi; Vidal-Alaball, Josep

doi:10.2196/39536

Cited by 11 publications

(12 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…At this same centre, convenience recruitment was carried out from 7 February 2022 to 31 May 2022. The study was explained and the information sheet and informed consent were given to all patients who came for a chest X-ray and met the inclusion criteria (33).…”

Section: Description Of the Study Population Time Frame And Data Coll...mentioning

confidence: 99%

See 1 more Smart Citation

External Validation of an Artificial Intelligence Algorithm for the Analysis of Chest X-Rays in the Primary Care Setting

Catalina,

Vidal-Alaball,

Fuster-Casanovas

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Interpreting chest X-rays is a complex task, and artificial intelligence algorithms for this purpose are currently being developed. It is important to perform external validations of these algorithms in order to implement them. This study therefore aims to externally validate an AI algorithm's diagnoses in real clinical practice, comparing them to a radiologist’s diagnoses. The aim is also to identify diagnoses the algorithm may not have been trained for. A prospective observational study for the external validation of the AI algorithm in a region of Catalonia, comparing the AI algorithm’s diagnosis with that of the reference radiologist, considered the gold standard. The external validation was performed with a sample of 278 images and reports, 51.8% of which showed no radiological abnormalities according to the radiologist's report. Analysing the validity of the AI algorithm, the average accuracy was 0.95 (95% CI 0.92; 0.98), the sensitivity was 0.48 (95% CI 0.30; 0.66) and the specificity was 0.98 (95% CI 0.97; 0.99). The conditions where the algorithm was most sensitive were external, upper abdominal and cardiac and/or valvular implants. On the other hand, the conditions where the algorithm was less sensitive were in the mediastinum, vessels and bone. The AI algorithm has proven to be useful when identifying images with or without conditions. However, in order to be a valuable tool to help and support experts, it requires additional real-world training to enhance its diagnostic capabilities for some of the conditions analysed.

show abstract

Section: Description Of the Study Population Time Frame And Data Coll...mentioning

confidence: 99%

“…Due to a problem with the image collection centre, the sample size calculated in the protocol (33) could not be reached. For this reason, the sample was recalculated by increasing the precision by one percentage point.…”

Section: Sample Sizementioning

confidence: 99%

External Validation of an Artificial Intelligence Algorithm for the Analysis of Chest X-Rays in the Primary Care Setting

Catalina,

Vidal-Alaball,

Fuster-Casanovas

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The diagnostic accuracy of the algorithms provided by the developers is quite high [ 7 , 8 , 9 ], reaching the same accuracy for radiologists [ 10 ], and for some solutions even exceeding them [ 11 , 12 ]. As of the beginning of 2023, 29 AI-based software products have European certification for medical use as a medical device (CE MDR/MDD), of which 11 have passed a similar certification in the United States [ 13 ].…”

Section: Introductionmentioning

confidence: 99%

“…It is important to note that one such software product, approved for circulation as a medical device, is intended for a completely autonomous analysis of CXR [ 14 ]. This AI algorithm sorts examinations, detects CXR without pathology, and forms a complete description protocol that does not require validation by a radiologist; this approach reduces the burden on the radiologist, allowing them to focus on cases with pathologies [ 11 ].…”

Section: Introductionmentioning

confidence: 99%

“…Will AI for CXR analysis also work well and demonstrate high diagnostic accuracy metrics in real clinical practice? There are very few large-scale works on this topic; it is possible to point out only a few studies [ 11 , 16 ]. Using recent research as an example, we can see the introduction of autonomous AI into clinical practice, but the issue of threshold values for diagnostic accuracy metrics remains open.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

AI-Based CXR First Reading: Current Limitations to Ensure Practical Value

Vasilev¹,

Vladzymyrskyy

Omelyanskaya³

et al. 2023

Diagnostics

View full text Add to dashboard Cite

We performed a multicenter external evaluation of the practical and clinical efficacy of a commercial AI algorithm for chest X-ray (CXR) analysis (Lunit INSIGHT CXR). A retrospective evaluation was performed with a multi-reader study. For a prospective evaluation, the AI model was run on CXR studies; the results were compared to the reports of 226 radiologists. In the multi-reader study, the area under the curve (AUC), sensitivity, and specificity of the AI were 0.94 (CI95%: 0.87–1.0), 0.9 (CI95%: 0.79–1.0), and 0.89 (CI95%: 0.79–0.98); the AUC, sensitivity, and specificity of the radiologists were 0.97 (CI95%: 0.94–1.0), 0.9 (CI95%: 0.79–1.0), and 0.95 (CI95%: 0.89–1.0). In most regions of the ROC curve, the AI performed a little worse or at the same level as an average human reader. The McNemar test showed no statistically significant differences between AI and radiologists. In the prospective study with 4752 cases, the AUC, sensitivity, and specificity of the AI were 0.84 (CI95%: 0.82–0.86), 0.77 (CI95%: 0.73–0.80), and 0.81 (CI95%: 0.80–0.82). Lower accuracy values obtained during the prospective validation were mainly associated with false-positive findings considered by experts to be clinically insignificant and the false-negative omission of human-reported “opacity”, “nodule”, and calcification. In a large-scale prospective validation of the commercial AI algorithm in clinical practice, lower sensitivity and specificity values were obtained compared to the prior retrospective evaluation of the data of the same population.

show abstract

Exploring the potential of artificial intelligence in improving skin lesion diagnosis in primary care

Escalé-Besa

Yélamos

Vidal‐Alaball

et al. 2023

Sci Rep

View full text Add to dashboard Cite

Dermatological conditions are a relevant health problem. Machine learning (ML) models are increasingly being applied to dermatology as a diagnostic decision support tool using image analysis, especially for skin cancer detection and disease classification. The objective of this study was to perform a prospective validation of an image analysis ML model, which is capable of screening 44 skin diseases, comparing its diagnostic accuracy with that of General Practitioners (GPs) and teledermatology (TD) dermatologists in a real-life setting. Prospective, diagnostic accuracy study including 100 consecutive patients with a skin problem who visited a participating GP in central Catalonia, Spain, between June 2021 and October 2021. The skin issue was first assessed by the GPs. Then an anonymised skin disease picture was taken and uploaded to the ML application, which returned a list with the Top-5 possible diagnosis in order of probability. The same image was then sent to a dermatologist via TD for diagnosis, as per clinical practice. The GPs Top-3, ML model’s Top-5 and dermatologist’s Top-3 assessments were compared to calculate the accuracy, sensitivity, specificity and diagnostic accuracy of the ML models. The overall Top-1 accuracy of the ML model (39%) was lower than that of GPs (64%) and dermatologists (72%). When the analysis was limited to the diagnoses on which the algorithm had been explicitly trained (n = 82), the balanced Top-1 accuracy of the ML model increased (48%) and in the Top-3 (75%) was comparable to the GPs Top-3 accuracy (76%). The Top-5 accuracy of the ML model (89%) was comparable to the dermatologist Top-3 accuracy (90%). For the different diseases, the sensitivity of the model (Top-3 87% and Top-5 96%) is higher than that of the clinicians (Top-3 GPs 76% and Top-3 dermatologists 84%) only in the benign tumour pathology group, being on the other hand the most prevalent category (n = 53). About the satisfaction of professionals, 92% of the GPs considered it as a useful diagnostic support tool (DST) for the differential diagnosis and in 60% of the cases as an aid in the final diagnosis of the skin lesion. The overall diagnostic accuracy of the model in this study, under real-life conditions, is lower than that of both GPs and dermatologists. This result aligns with the findings of few existing prospective studies conducted under real-life conditions. The outcomes emphasize the significance of involving clinicians in the training of the model and the capability of ML models to assist GPs, particularly in differential diagnosis. Nevertheless, external testing in real-life conditions is crucial for data validation and regulation of these AI diagnostic models before they can be used in primary care.

show abstract

Developing an Artificial Intelligence Model for Reading Chest X-rays: Protocol for a Prospective Validation Study

Cited by 11 publications

References 19 publications

External Validation of an Artificial Intelligence Algorithm for the Analysis of Chest X-Rays in the Primary Care Setting

External Validation of an Artificial Intelligence Algorithm for the Analysis of Chest X-Rays in the Primary Care Setting

AI-Based CXR First Reading: Current Limitations to Ensure Practical Value

Exploring the potential of artificial intelligence in improving skin lesion diagnosis in primary care

Contact Info

Product

Resources

About