Purpose:To assess interpretation performance and radiation dose when two-dimensional synthesized mammography (SM) images versus standard full-field digital mammography (FFDM) images are used alone or in combination with digital breast tomosynthesis images. Materials and Methods:A fully crossed, mode-balanced multicase (n = 123), multireader (n = 8), retrospective observer performance study was performed by using deidentified images acquired between 2008 and 2011 with institutional review board approved, HIPAA-compliant protocols, during which each patient signed informed consent. The cohort included 36 cases of biopsy-proven cancer, 35 cases of biopsy-proven benign lesions, and 52 normal or benign cases (Breast Imaging Reporting and Data System [BI-RADS] score of 1 or 2) with negative 1-year follow-up results. Accuracy of sequentially reported probability of malignancy ratings and seven-category forced BI-RADS ratings was evaluated by using areas under the receiver operating characteristic curve (AUCs) in the random-reader analysis. Results:Probability of malignancy-based mean AUCs for SM and FFDM images alone was 0.894 and 0.889, respectively (difference, 20.005; 95% confidence interval [CI]: 20.062, 0.054; P = .85). Mean AUC for SM with tomosynthesis and FFDM with tomosynthesis was 0.916 and 0.939, respectively (difference, 0.023; 95% CI: 20.011, 0.057; P = .19). In terms of the reader-specific AUCs, five readers performed better with SM alone versus FFDM alone, and all eight readers performed better with combined FFDM and tomosynthesis (absolute differences from 0.003 to 0.052). Similar results were obtained by using a nonparametric analysis of forced BI-RADS ratings.

Rationale and Objectives The “binormal” model is the most frequently used tool for parametric receiver operating characteristic (ROC) analysis. The binormal ROC curves can have “improper” (non-concave) shapes that are unrealistic in many practical applications, and several tools (eg, PROPROC) have been developed to address this problem. However, due to the general robustness of binormal ROCs, the improperness of the fitted curves might carry little consequence for inferences about global summary indices, such as the area under the ROC curve (AUC). In this work, we investigate the effect of severe improperness of fitted binormal ROC curves on the reliability of AUC estimates when the data arise from an actually proper curve. Materials and Methods We designed theoretically proper ROC scenarios that induce severely improper shape of fitted binormal curves in the presence of well-distributed empirical ROC points. The binormal curves were fitted using maximum likelihood approach. Using simulations, we estimated the frequency of severely improper fitted curves, bias of the estimated AUC, and coverage of 95% confidence intervals (CIs). In Appendix S1, we provide additional information on percentiles of the distribution of AUC estimates and bias when estimating partial AUCs. We also compared the results to a reference standard provided by empirical estimates obtained from continuous data. Results We observed up to 96% of severely improper curves depending on the scenario in question. The bias in the binormal AUC estimates was very small and the coverage of the CIs was close to nominal, whereas the estimates of partial AUC were biased upward in the high specificity range and downward in the low specificity range. Compared to a non-parametric approach, the binormal model led to slightly more variable AUC estimates, but at the same time to CIs with more appropriate coverage. Conclusions The improper shape of the fitted binormal curve, by itself, ie, in the presence of a sufficient number of well-distributed points, does not imply unreliable AUC-based inferences.

Receiver Operating Characteristic (ROC) analysis provides an important methodology for assessing traditional (e.g., imaging technologies and clinical practices) and new (e.g., genomic studies, biomarker development) diagnostic problems. The area under the clinically/practically relevant part of the ROC curve (partial area or pAUC) is an important performance index summarizing diagnostic accuracy at multiple operating points (decision thresholds) that are relevant to actual clinical practice. A robust estimate of the pAUC is provided by the area under the corresponding part of the empirical ROC curve. We derive a closed-form expression for the jackknife variance of the partial area under the empirical ROC curve. Using the derived analytical expression we investigate the differences between the jackknife variances and a conventional variance estimator. The relative properties in finite samples are demonstrated in a simulation study. The developed formula enables an easy way to estimate the variance of the empirical pAUC, thereby substantially reducing the computation burden, and provides important insight into the structure of the variability. We demonstrate that when compared with the conventional approach, the jackknife variance has negligible bias, and leads to a more appropriate type I error rate of the Wald-type test. The use of the jackknife variance is illustrated in the analysis of a dataset from a diagnostic imaging study.