Addressing the Challenge of Assessing Physician-Level Screening Performance: Mammography as an Example

Abstract

BackgroundMotivated by the challenges in assessing physician-level cancer screening performance and the negative impact of misclassification, we propose a method (using mammography as an example) that enables confident assertion of adequate or inadequate performance or alternatively recognizes when more data is required.MethodsUsing established metrics for mammography screening performance–cancer detection rate (CDR) and recall rate (RR)–and observed benchmarks from the Breast Cancer Surveillance Consortium (BCSC), we calculate the minimum volume required to be 95% confident that a physician is performing at or above benchmark thresholds. We graphically display the minimum observed CDR and RR values required to confidently assert adequate performance over a range of interpretive volumes. We use a prospectively collected database of consecutive mammograms from a clinical screening program outside the BCSC to illustrate how this method classifies individual physician performance as volume accrues.ResultsOur analysis reveals that an annual interpretive volume of 2770 screening mammograms, above the United States’ (US) mandatory (480) and average (1777) annual volumes but below England’s mandatory (5000) annual volume is necessary to confidently assert that a physician performed adequately. In our analyzed US practice, a single year of data uniformly allowed confident assertion of adequate performance in terms of RR but not CDR, which required aggregation of data across more than one year.ConclusionFor individual physician quality assessment in cancer screening programs that target low incidence populations, considering imprecision in observed performance metrics due to small numbers of patients with cancer is important.

Association between Screening Mammography Recall Rate and Interval Cancers in the UK Breast Cancer Service Screening Program: A Cohort Study

“…RAR is rapidly available but ICR entails long reporting delays (22). Unlike RAR, the ICR, CDR, and positive predictive value all require large numbers to be reliable (30). Our demonstration of the association between increasing RAR and decreasing ICR supports the possibility of identifying a minimum RAR as a benchmark that is immediately available and statistically stable.…”

Test performance metrics for breast, cervical, colon, and lung cancer screening: a systematic review

“…Performance metrics requiring linkage to cancer outcomes and with very low incidence, such as colonoscopy-related mortality, are inherently more challenging to meaningfully implement. Ideally, quality metrics should be practical to measure and usable for monitoring screening quality and/or to improve care—ideally in as close to real-time as possible, such as the demonstration of associations between the recall rate for mammography and postscreening cancer rates ( 31 ). For example, individual radiologists, provided they read more than 2700 mammograms per year, can receive robust estimates of their performance as compared with established benchmarks, though implementation in the United States remains incomplete ( 32 ).…”

Criteria for Identifying Radiologists With Acceptable Screening Mammography Interpretive Performance on Basis of Multiple Performance Measures

“…A limitation of our study is that our methods do not account for variability in the observed measures due to small sample sizes and rare events. [27] We included both digital and film mammography exams in our normative data; however, most mammograms in the US are now digital. We do not think this influenced our criteria, because interpretative performance of digital mammography is similar to that of film mammography except in some subsets of women [28–30].…”