Changes in cancer detection and false-positive recall in mammography using artificial intelligence: a retrospective, multireader study

Abstract

Background Mammography is the current standard for breast cancer screening. This study aimed to develop an artificial intelligence (AI) algorithm for diagnosis of breast cancer in mammography, and explore whether it could benefit radiologists by improving accuracy of diagnosis. Methods In this retrospective study, an AI algorithm was developed and validated with 170 230 mammography examinations collected from five institutions in South Korea, the USA, and the UK, including 36 468 cancer positive confirmed by biopsy, 59 544 benign confirmed by biopsy (8827 mammograms) or follow-up imaging (50 717 mammograms), and 74 218 normal. For the multicentre, observer-blinded, reader study, 320 mammograms (160 cancer positive, 64 benign, 96 normal) were independently obtained from two institutions. 14 radiologists participated as readers and assessed each mammogram in terms of likelihood of malignancy (LOM), location of malignancy, and necessity to recall the patient, first without and then with assistance of the AI algorithm. The performance of AI and radiologists was evaluated in terms of LOM-based area under the receiver operating characteristic curve (AUROC) and recall-based sensitivity and specificity. Findings The AI standalone performance was AUROC 0•959 (95% CI 0•952-0•966) overall, and 0•970 (0•963-0•978) in the South Korea dataset, 0•953 (0•938-0•968) in the USA dataset, and 0•938 (0•918-0•958) in the UK dataset. In the reader study, the performance level of AI was 0•940 (0•915-0•965), significantly higher than that of the radiologists without AI assistance (0•810, 95% CI 0•770-0•850; p<0•0001). With the assistance of AI, radiologists' performance was improved to 0•881 (0•850-0•911; p<0•0001). AI was more sensitive to detect cancers with mass (53 [90%] vs 46 [78%] of 59 cancers detected; p=0•044) or distortion or asymmetry (18 [90%] vs ten [50%] of 20 cancers detected; p=0•023) than radiologists. AI was better in detection of T1 cancers (73 [91%] vs 59 [74%] of 80; p=0•0039) or node-negative cancers (104 [87%] vs 88 [74%] of 119; p=0•0025) than radiologists. Interpretation The AI algorithm developed with large-scale mammography data showed better diagnostic performance in breast cancer detection compared with radiologists. The significant improvement in radiologists' performance when aided by AI supports application of AI to mammograms as a diagnostic support tool.

Evaluating AI in breast cancer screening: a complex task

“…For example, the specificity scores shown by radiologists, AI-aided radiologists, and standalone AI are all much lower than what is normally seen in screening. 10 This is the main limitation of the study, which is a limitation shared with most other studies on AI in breast cancer screening. Thus, while promising algorithms, such as the one evaluated by Kim and colleagues, could potentially improve breast screening, to prove this it is imperative to conduct retrospective studies on unaltered screening materials and, even more importantly, randomised prospective trials.…”

“…Nonenriched screening material has 0·5-1·0% cancer prevalence, whereas the enriched material used by Kim and colleagues had 50% prevalence. 10 The performance of radiologists in a real-world screening situation is not necessarily, or even likely to be, the same as it would be on the material evaluated in Kim and colleagues' reader study. For example, the specificity scores shown by radiologists, AI-aided radiologists, and standalone AI are all much lower than what is normally seen in screening.…”

“…[6][7][8][9] In The Lancet Digital Health, Hyo-Eun Kim and colleagues' multireader study provides new insights in the field. 10 In short, the South Korean academic-industrial partnership behind the study trained, validated, and tested a promising AI algorithm to aid detection of breast cancer in mammography. It was trained on a large set of images (n=170 230 mammograms) from three different countries and at least three different manufacturers.…”

Leveraging Deep Learning to Simulate Coronavirus Spike proteins has the potential to predict future Zoonotic sequences

“…Recently, uses for DL have been described in health care, particularly in screening for breast cancer 17,18 and for use with electrocardiogram (ECG) traces 19,20 . Potential novel antibiotics were searched out by screening known drug databases for structures 21 .…”

Deep Neural Network for Predicting Diabetic Retinopathy from Risk Factors

“…Therefore, to identify prediction of positive cases (retinopathy) accurately, the model should focus more on recall rather than precision. In the medical area, it is common to focus more on sensitivity (recall) and specificity to evaluate medical tests [46]. Furthermore, it is also important to detect positive cases accurately, since when the model fails to detect the retinopathy, it will lead to blindness in such patients.…”