Evaluating the effectiveness of abbreviated breast MRI (abMRI) interpretation training for mammogram readers: a multi-centre study assessing diagnostic performance, using an enriched dataset

Abstract

Background Abbreviated breast MRI (abMRI) is being introduced in breast screening trials and clinical practice, particularly for women with dense breasts. Upscaling abMRI provision requires the workforce of mammogram readers to learn to effectively interpret abMRI. The purpose of this study was to examine the diagnostic accuracy of mammogram readers to interpret abMRI after a single day of standardised small-group training and to compare diagnostic performance of mammogram readers experienced in full-protocol breast MRI (fpMRI) interpretation (Group 1) with that of those without fpMRI interpretation experience (Group 2). Methods Mammogram readers were recruited from six NHS Breast Screening Programme sites. Small-group hands-on workstation training was provided, with subsequent prospective, independent, blinded interpretation of an enriched dataset with known outcome. A simplified form of abMRI (first post-contrast subtracted images (FAST MRI), displayed as maximum-intensity projection (MIP) and subtracted slice stack) was used. Per-breast and per-lesion diagnostic accuracy analysis was undertaken, with comparison across groups, and double-reading simulation of a consecutive screening subset. Results 37 readers (Group 1: 17, Group 2: 20) completed the reading task of 125 scans (250 breasts) (total = 9250 reads). Overall sensitivity was 86% (95% confidence interval (CI) 84–87%; 1776/2072) and specificity 86% (95%CI 85–86%; 6140/7178). Group 1 showed significantly higher sensitivity (843/952; 89%; 95%CI 86–91%) and higher specificity (2957/3298; 90%; 95%CI 89–91%) than Group 2 (sensitivity = 83%; 95%CI 81–85% (933/1120) p < 0.0001; specificity = 82%; 95%CI 81–83% (3183/3880) p < 0.0001). Inter-reader agreement was higher for Group 1 (kappa = 0.73; 95%CI 0.68–0.79) than for Group 2 (kappa = 0.51; 95%CI 0.45–0.56). Specificity improved for Group 2, from the first 55 cases (81%) to the remaining 70 (83%) (p = 0.02) but not for Group 1 (90–89% p = 0.44), whereas sensitivity remained consistent for both Group 1 (88–89%) and Group 2 (83–84%). Conclusions Single-day abMRI interpretation training for mammogram readers achieved an overall diagnostic performance within benchmarks published for fpMRI but was insufficient for diagnostic accuracy of mammogram readers new to breast MRI to match that of experienced fpMRI readers. Novice MRI reader performance improved during the reading task, suggesting that additional training could further narrow this performance gap.

Optimising the diagnostic accuracy of First post-contrAst SubtracTed breast MRI (FAST MRI) through interpretation-training: a multicentre e-learning study, mapping the learning curve of NHS Breast Screening Programme (NHSBSP) mammogram readers using an enriched dataset

“…Following a single day's training, mammogram readers achieved an overall sensitivity of 86% and speci city of 86%. However, the diagnostic accuracy achieved in the study by those with previous experience of reporting fpMRI (sensitivity 89%, speci city 90%) remained higher than for those with no such previous experience (sensitivity 83% p < 0.001, speci city 82% p < 0.001) (11).…”

“…Whilst the reader agreement with the true outcome (kappa) and the DOR did not differ signi cantly between the readers who had previously attended in-person FAST MRI interpretation-training (11/22 in Group 1 and 7/21 in Group 2) and those who had not, the sensitivity for cancer detection was higher and the speci city lower for the "attended" cohort than for the "not attended" cohort. Looking at the individual performance, during a previous study (11), of the 14 participants of the current study who had attended previous small group training, 8 of these participants had a sensitivity in the top 9 sensitivities of participants in the previous study and none were in the bottom 7 sensitivities (11). Additionally, 8 of these participants had speci city in the bottom 12 for speci city in the previous study and 3 were in the top 11 speci cities (11).…”

“…Previously developed software (RiViewer), that displays FAST MRI as maximum intensity projection (MIP) and stacked, subtracted slices, was used in which biopsy-proven cancers had been drawn onto images electronically as regions of interest (ROI volumes) to provide ground truth. As in a previous study (11), during hands-on workstation training, learners reviewed 29 training FAST MRI scans, and could discover the ground truth at the touch of a button.…”

“…Looking at the individual performance, during a previous study (11), of the 14 participants of the current study who had attended previous small group training, 8 of these participants had a sensitivity in the top 9 sensitivities of participants in the previous study and none were in the bottom 7 sensitivities (11). Additionally, 8 of these participants had speci city in the bottom 12 for speci city in the previous study and 3 were in the top 11 speci cities (11). Therefore, self-selection bias could have contributed to the within group signi cant differences of sensitivity and speci city found for attendance vs. non-attendance at previous in person training.…”

“…Formative assessment is an educational technique where the assessment process includes feedback to the learner so that in addition to measuring the learner's achievement, it enhances learning(18). We hypothesised that readers' diagnostic accuracy could be optimised by converting summative assessment (without any feedback to the readers), as used in previous FAST MRI reader training studies (11,13) into formative assessment (by giving readers immediate feedback for each FAST MRI scan read during the assessment task).…”