Correlation Between Screening Mammography Interpretive Performance on a Test Set and Performance in Clinical Practice

Abstract

Rationale and Objectives Evidence is inconsistent about whether radiologists' interpretive performance on a screening mammography test set reflects their performance in clinical practice. This study aimed to estimate the correlation between test-set and clinical performance and determine if the correlation is influenced by cancer prevalence or lesion difficulty in the test set. Materials and Methods This institutional review board–approved study randomized 83 radiologists from six Breast Cancer Surveillance Consortium registries to assess one of four test sets of 109 screening mammograms each; 48 radiologists completed a fifth test set of 110 mammograms 2 years later. Test sets differed in number of cancer cases and difficulty of lesion detection. Test-set sensitivity and specificity were estimated using woman-level and breast-level recall with cancer status and expert opinion as gold standards. Clinical performance was estimated using women-level recall with cancer status as the gold standard. Spearman rank correlations between test-set and clinical performance with 95% confidence intervals (CI) were estimated. Results For test sets with fewer cancers (N=15) that were more difficult to detect, correlations were weak-to-moderate for sensitivity (woman-level=0.46, 95%CI=0.16, 0.69; breast-level=0.35, 95%CI=0.03, 0.61) and weak for specificity (0.24, 95%CI=0.01, 0.45) relative to expert recall. Correlations for test sets with more cancers (N=30) were close to zero and not statistically significant. Conclusions Correlations between screening performance on a test set and performance in clinical practice are not strong. Test set performance more accurately reflects performance in clinical practice if cancer prevalence is low and lesions are challenging to detect.

Fatigue and vigilance in medical experts detecting breast cancer

“…However, these studies lack ecological validity because participants know that they are taking part in research so the jeopardy of missing a cancer or allowing dangerous items onto an aeroplane is absent. Indeed, there is evidence from medical imaging that performance during experimental studies is not reflective of performance in clinical practice (the laboratory effect) ( 26 – 28 ). Despite the limitations of current research, procedures aimed at reducing vigilance decrements have been developed and implemented ( 29 – 31 ).…”