Retrospective large-scale evaluation of an AI system as an independent reader for double reading in breast cancer screening

Abstract

Screening mammography with two human readers increases cancer detection and lowers recall rates, but high resource requirements and a shortage of qualified readers make double reading unsustainable in many countries. The use of AI as an independent reader may yield more objective, accurate and outcome-based screening. Clinical validation of AI requires large-scale, multi-site, multi-vendor studies on unenriched cohorts.This retrospective study evaluated the performance of the Mia™ version 2.0.1 AI system from Kheiron Medical Technologies on an unenriched sample (275,900 cases from 177,882 participants) collected across seven screening sites in two countries and four hardware vendors, and is representative of a real-world screening population over 10 years. Performance was determined for standalone AI and simulated double reading to assess non-inferiority and superiority on relevant screening metrics.Standalone AI showed superiority on sensitivity and non-inferiority on specificity while detecting 29.7% of cancers found within three years after screening, and 29.8% of missed interval cancers. Double reading with AI was at least non-inferior compared to human double reading at every metric, with superiority for recall rate, specificity and positive predictive value (PPV). AI as an independent reader reduced the workload, but increased arbitration rate from 3.3% to 12.3%. Applying the AI system under investigation would have reduced the overall number of human reads required by 44.8%. The recall rate was reduced by a relative 4.1%, suggesting there could be fewer follow-up procedures, reduced stress for patients, and less administrative and clinical work.Using the AI system as an independent reader maintains the standard of care of double reading, detects cancers missed by human readers, while automating a substantial part of the workflow, and could therefore bring significant clinical and operational benefits.

Use of artificial intelligence for image analysis in breast cancer screening programmes: systematic review of test accuracy

“…In two of the largest retrospective cohort studies of AI to replace radiologists in Europe (n=76 813 women),3536 all AI systems were less accurate than consensus of two radiologists, and 34 of 36 AI systems were less accurate than a single reader. One unpublished study is in line with these findings 40. This large retrospective study (n=275 900 women) reported higher sensitivity of AI in comparison with the original first reader decision but lower specificity, and the AI system was less accurate than consensus reading 40.…”

“…One unpublished study is in line with these findings. 40 This large retrospective study (n=275 900 women) reported higher sensitivity of AI in comparison with the original first reader decision but lower specificity, and the AI system was less accurate than consensus reading. 40 Four retrospective studies 25 26 27 31 indicated that at lower thresholds, AI can achieve high sensitivity so might be suitable for triaging which women should receive radiological review.…”

“…One unpublished study is in line with these findings. 40 This large retrospective study (n=275 900 women) reported higher sensitivity of AI in comparison with the original first reader decision but lower specificity, and the AI system was less Reference standard for test negatives was double reading not follow-up. (A) Dembrower 2020a: retrospective study using AI (Lunit version 5.5.0.16) for pre-screen (point estimates not based on exact numbers).…”

“…Radiologist double reading for this cohort would be 100% specificity and 0% sensitivity as this was only in a cohort of women with screen (true and false) negative mammograms accurate than consensus reading. 40 Four retrospective studies 25-27 31 indicated that at lower thresholds, AI can achieve high sensitivity so might be suitable for triaging which women should receive radiological review. Further research is required to determine the most appropriate threshold as the only study which prespecified the threshold for triage achieved 88.5% sensitivity.…”

A novel workflow for the safe and effective integration of AI as supporting reader in double reading breast cancer screening: A large-scale retrospective evaluation

“…A novel workflow of using AI as supporting reader (AI-SR) was simulated and compared to the historical human double reading (HDR) and AI serving as an independent second reader (AI-IR) (see Figure 1). AI-SR is a variation of the previously explored AI-IR workflow [3,6], specifically designed to avoid an increase in arbitration rates. All performance comparisons were determined on the same unenriched screening cohorts representative of data the AI would see in real-world deployments.…”