Proper receiver operating characteristic analysis: The bigamma model

Dorfman, Donald D.; Berbaum, Kevin S.; Metz, Charles E.; Lenth, Russell V.; Hanley, James A.; Dagga, Hatem Abu

doi:10.1016/s1076-6332(97)80013-x

Cited by 113 publications

(86 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A more recent development of the maximum likelihood estimation for the proper ROC analysis (14,15) may alleviate some of the problems caused by unevenly distributed operating points. It will be of interest to investigate if the discrete rating scales with different number of categories may influence the proper ROC analysis in future studies.…”

Section: Discussionmentioning

confidence: 99%

Quasi-Continuous and Discrete Confidence Rating Scales for Observer Performance Studies

et al. 2007

View full text Add to dashboard Cite

Rationale and Objectives-To examine the effects of the number of categories in the rating scale used in an observer experiment on the results of ROC analysis by a simulation study.Materials and Methods-We have previously evaluated the effects of computer-aided diagnosis (CAD) on radiologists' characterization of malignant and benign breast masses in serial mammograms. The evaluation of the likelihood of malignancy was performed on a quasi-continuous (0-100 points) confidence-rating scale. In this study, we simulated the use of discrete confidencerating scales with fewer number of categories and analyzed the results with receiver operating characteristic (ROC) methodology. The observers' estimates of the likelihood of malignancy were also mapped to BI-RADS assessments with 5 and 7 categories and ROC analysis was performed. The area under the ROC curve and the partial area index obtained from ROC analysis of the different confidence-rating scales were compared.Results-The fitted ROC curves and the performance indices do not change significantly when the confidence-rating scales were varied from 6 to 101 points if the estimated operating points obtained directly from the data are distributed relatively evenly over the entire range of true-positive fraction (TPF) and false-positive fraction (FPF). The mapping of the likelihood of malignancy observer data to the 7-category BI-RADS assessment scale allowed reliable ROC analysis, whereas mapping to the 5-category BI-RADS scale could cause erratic ROC curve fitting because of the lack of operating points in the mid-range or failure in ROC curve fitting because of data degeneration for some observers.Conclusion-ROC analysis of discrete confidence rating scales with few but relatively evenly distributed data points over the entire FPF and TPF range is comparable to that of a quasi-continuous rating scale. However, ROC analysis of discrete confidence rating scales with few and unevenly distributed data points may cause unreliable estimations.

show abstract

Section: Discussionmentioning

confidence: 99%

Quasi-Continuous and Discrete Confidence Rating Scales for Observer Performance Studies

et al. 2007

View full text Add to dashboard Cite

show abstract

“…We applied a nonlinear normality Box-Cox transformation algorithm (see Appendix A.1) (2-4,14). After such a parametric transformation, a mixture of two normal distributions can be assumed, known as the bi-normal model in the literature (1,15,16). We assume here that the measurements in class C 0 have a standard normal distribution, while the measurements C 1 of the diseased class have a normal distributions.…”

Section: Methodsmentioning

confidence: 99%

Statistical validation based on parametric receiver operating characteristic analysis of continuous classification data1

Zou¹,

Warfield²,

Tempany³

et al. 2003

Academic Radiology

View full text Add to dashboard Cite

Rationale and Objectives-The accuracy of diagnostic test and imaging segmentation is important in clinical practice because it has a direct impact on therapeutic planning. Statistical validations of classification accuracy was conducted based on parametric receiver operating characteristic analysis, illustrated on three radiologic examples.Materials and Methods-Two parametric models were developed for diagnostic or imaging data. Example 1: A semiautomated fractional segmentation algorithm was applied to magnetic resonance imaging of nine cases of brain tumors. The tumor and background pixel data were assumed to have bi-beta distributions. Fractional segmentation was validated against an estimated composite pixelwise gold standard based on multi-reader manual segmentations. Example 2: The predictive value of 100 cases of spiral computed tomography of ureteral stone sizes, distributed as bi-normal after a nonlinear transformation, under two treatment options received. Example 3: One hundred eighty cases had prostate-specific antigen levels measured in a prospective clinical trial. Radical prostatectomy was performed in all to provide a binary gold standard of local and advanced cancer stages. Prostate-specific antigen level was transformed and modeled by bi-normal distributions. In all examples, areas under the receiver operating characteristic curves were computed. Conclusion-All clinical examples yielded fair to excellent accuracy. The validation metric area under the receiver operating characteristic curves may be generalized to evaluating the performances of several continuous classifiers related to imaging. Results-The KeywordsBrain segmentation; magnetic resonance; prostate specific antigen (PSA); genitourinary system; computed tomography; receiver operating characteristic (ROC) analysisThe accuracy of diagnostic test and imaging segmentation is important in clinical practice because it has a direct impact on therapeutic planning. Recently, continuous classification tools In contrast, traditional diagnostic tests were often based on an ordinal rating scale. For example, a five-point scale might be adopted for observer performance evaluations, where 1 = definitely normal, 2 = probably normal, 3 = probably abnormal, 4 = probably abnormal, and 5 = definitely abnormal. A discrete subjective rating method was used in a multi-modal (magnetic resonance [MR], computed tomography [CT], and ultrasound) comparative ovarian cancer technology assessment study (5,6), in one of a series of prospective multicenter Radiologic Diagnostic Oncology Group clinical trials sponsored by the funded by the National Institutes of Health in the 1990s. The advantages of the continuous diagnostic over ordinal scale are that detailed information is preserved, they are more natural with the advancements in measurement tools and computing methods, and enable more objective interpretations. Ordinal rating data will not be the focus of this article. Instead, we will evaluate the performances of continuous classifiers only.To conduct a...

show abstract

“…Such curves are TPP-asymmetric (as described in Figure 1). For example, see Dorfman et al [8]. If r = 1, f 1 (x) and f 2 (x) are the same as for the g (the dashed line).…”

Section: The Constant-shape Bi-gamma Roc Curvementioning

confidence: 97%

“…ROC curves that are monotone increasing above the main diagonal of the plot over the whole domain are sometimes referred to as "proper" ROC curves (see, e.g., Section 4.6 in [1]). Some continuous parametric ROC curves are proper, some are not; for example, it is well-known that the bi-Normal ROC curve is not in general proper, while the bi-gamma ROC curve is proper [8].…”

Section: Introductionmentioning

confidence: 99%

Symmetry Properties of Bi-Normal and Bi-Gamma Receiver Operating Characteristic Curves are Described by Kullback-Leibler Divergences

Hughes

Bhattacharya

2013

Entropy

View full text Add to dashboard Cite

Receiver operating characteristic (ROC) curves have application in analysis of the performance of diagnostic indicators used in the assessment of disease risk in clinical and veterinary medicine and in crop protection. For a binary indicator, an ROC curve summarizes the two distributions of risk scores obtained by retrospectively categorizing subjects as cases or controls using a gold standard. An ROC curve may be symmetric about the negative diagonal of the graphical plot, or skewed towards the left-hand axis or the upper axis of the plot. ROC curves with different symmetry properties may have the same area under the curve. Here, we characterize the symmetry properties of bi-Normal and bi-gamma ROC curves in terms of the Kullback-Leibler divergences (KLDs) between the case and control distributions of risk scores. The KLDs describe the known symmetry properties of bi-Normal ROC curves, and newly characterize the symmetry properties of constant-shape and constant-scale bi-gamma ROC curves. It is also of interest to note an application of KLDs where their asymmetry-often an inconvenience-has a useful interpretation.

show abstract

Proper receiver operating characteristic analysis: The bigamma model

Cited by 113 publications

References 16 publications

Quasi-Continuous and Discrete Confidence Rating Scales for Observer Performance Studies

Quasi-Continuous and Discrete Confidence Rating Scales for Observer Performance Studies

Statistical validation based on parametric receiver operating characteristic analysis of continuous classification data1

Symmetry Properties of Bi-Normal and Bi-Gamma Receiver Operating Characteristic Curves are Described by Kullback-Leibler Divergences

Contact Info

Product

Resources

About