Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals—Rating-method data

Dorfman, Donald D.; Alf, Edward F.

doi:10.1016/0022-2496(69)90019-4

Cited by 812 publications

(428 citation statements)

References 6 publications

Supporting

Mentioning

422

Contrasting

Unclassified

Order By: Relevance

“…The theoretical ROC curve for continuously distributed data was obtained by maximum likelihood estimation. 35,36 The AUC values for the selected model and full model were almost the same (0.8739 and 0.8748, respectively) and were relatively high, suggesting that the accuracy of the two models was almost the same and that subjects with high values of serum total bilirubin level could be distinguished by the three htSNPs within the UGT1A1 gene, gender and age. In this study, we genotyped only five SNPs in the UGT1A1 gene.…”

Section: Discussionmentioning

confidence: 89%

Association study between single-nucleotide polymorphisms in 199 drug-related genes and commonly measured quantitative traits of 752 healthy Japanese subjects

2009

View full text Add to dashboard Cite

With dense single-nucleotide polymorphism (SNP) maps for 199 drug-related genes, we examined associations between 4190 SNPs and 38 commonly measured quantitative traits using data from 752 healthy Japanese subjects. On analysis, we observed a strong association between five SNPs within the uridine diphosphate glucuronosyltransferase 1A1 (UGT1A1) gene and serum total bilirubin levels (minimum P-value in Mann-Whitney test¼1.82Â10 10 ). UGT1A1 catalyzes the conjugation of bilirubin with glucuronic acid, thus enhancing bilirubin elimination. This enzyme is known to play an important role in the variation of serum bilirubin levels. The five SNPs, including a nonsynonymous SNP-rs4148323 (211G4A or G71R variant allele known as UGT1A1*6)-showed strong linkage disequilibrium with each other. No other genes were clearly associated with serum total bilirubin levels. Results of linear multiple regression analysis on serum total bilirubin levels followed by analysis of variance showed that at least 13% of the variance in serum total bilirubin levels could be explained by three haplotype-tagging SNPs in the UGT1A1 gene.

show abstract

Section: Discussionmentioning

confidence: 89%

Association study between single-nucleotide polymorphisms in 199 drug-related genes and commonly measured quantitative traits of 752 healthy Japanese subjects

2009

View full text Add to dashboard Cite

show abstract

“…The ROC curve will then have a n e x p o n e n tial Draft Only 7 form and can be tted either: directly using an iterative M a x i m um Likelihood (ML) estimation (4) , giving the di erence in means and the ratio of the variances of the positive and negative distributions or, if the ROC curve is plotted on double probability paper, a straight line can be tted to the points on the ROC curve (5) . The slope and intercept of this tted line are then used to obtain an estimate of the AUC.…”

Section: Auc As a Performance Measurementioning

confidence: 99%

The use of the area under the ROC curve in the evaluation of machine learning algorithms

Bradley

1997

Pattern Recognition

5,603

2,981

View full text Add to dashboard Cite

SummaryIn this paper we i n vestigate the use of receiver operating characteristic (ROC) curve f o r the evaluation of machine learning algorithms. In particular, we i n vestigate the use of the area under the ROC curve ( A UC) as a measure of classi er performance. The machine learning algorithms used are chosen to be representative of those in common use: two decision trees (C4.5 and Multiscale Classi er) two n e u r a l n e t works (Perceptron and Multi-layer Perceptron) and two statistical methods (K-Nearest Neighbours and a Quadratic Discriminant F unction).The evaluation is done using six, \real world," medical diagnostics data sets that contain a varying numbers of inputs and samples, but are primarily continuous input, binary classi cation problems. We i d e n tify three forms of bias that can a ect comparisons of this type (estimation, selection, and expert bias) and detail the methods used to avoid them. We compare and discuss the use of AUC with the conventional measure of classi er performance, overall accuracy (the probability of a correct response). It is found that AUC exhibits a number of desirable properties when compared to overall accuracy: increased sensitivity in Analysis of Variance (ANOVA) tests a standard error that decreased as both AUC and the number of test samples increased decision threshold independent invariant t o a priori class probabilities and it gives an indication of the amount o f \ w ork done" by a classi cation scheme, giving low scores to both random and \one class only" classi ers.It has been known for some time that AUC actually represents the probability that a randomly chosen positive example is correctly rated (ranked) with greater suspicion than a randomly chosen negative example. Moreover, this probability of correct ranking is the same quantity estimated by the non-parametric Wilcoxon statistic. We use this equivalence to show that the standard deviation of AUC, estimated using 10 fold cross validation, is a reliable estimator of the standard error estimated using the Wilcoxon test. The paper concludes with the recommendation that AUC be used in preference to overall accuracy when \single number" evaluation of machine learning algorithms is required. Draft Only 3 AbstractIn this paper we i n vestigate the use of the area under the receiver operating characteristic (ROC) curve ( A UC) as a performance measure for machine learning algorithms.As a case study we e v aluate six machine learning algorithms (C4.5, Multiscale Classi er, Perceptron, Multi-layer Perceptron, K-Nearest Neighbours, and a Quadratic Discriminant F unction) on six \real world" medical diagnostics data sets. We compare and discuss the use of AUC to the more conventional overall accuracy and nd that AUC exhibits a number of desirable properties when compared to overall accuracy: increased sensitivity in Analysis of Variance (ANOVA) tests a standard error that decreased as both AUC and the number of test samples increased decision threshold independent and it is invariant t o a priori class proba...

show abstract

“…At the 5% significance level, if p < 0.05, then one rejects the null hypothesis, i.e., the fit is not good. In practice one often accepts p-values as small as 0.001 as evidence of a reasonable fit (34,35).…”

Section: Validity Of the Modelmentioning

confidence: 99%

Spatial Localization Accuracy of Radiologists in Free-Response Studies: Inferring Perceptual FROC Curves from Mark-Rating Data

2007

View full text Add to dashboard Cite

Free-response data consists of a set of mark-ratings pairs. Prior to analysis the data is classified or "scored" into lesion and non-lesion localizations. The scoring is done by choosing an acceptanceradius and classifying marks within the acceptance-radius of lesion centers as lesion localizations, and all other marks are classified as non-lesion localizations. The scored data is plotted as a freeresponse receiver operating characteristic (FROC) curve, essentially a plot of appropriately normalized numbers of lesion localizations vs. non-lesion localizations. Scored FROC curves are frequently used to compare imaging systems and computer aided detection (CAD) algorithms. However, the choice of acceptance-radius is arbitrary. This makes it difficult to compare curves from different studies and to estimate true performance. To resolve this issue the concept of two types of marks is introduced: perceptual hits and perceptual misses. A perceptual hit is a mark made in response to the observer seeing the lesion. A perceptual miss is a mark made in response to the observer seeing a (lesion-like) non-lesion. A method of estimating the most probable numbers of perceptual hits and misses is described. This allows one to plot a perceptual FROC operating point and by extension a perceptual FROC curve. Unlike a scored FROC operating point, a perceptual point is independent of the choice of acceptance-radius. The method does not allow one to identify individual marks as perceptual hits or misses -only the most probable numbers. It is based on a 3-parameter statistical model of the spatial distributions of perceptual hits and misses relative to lesion centers. The method has been applied to an observer dataset in which mammographers, and residents with different levels of experience were asked to locate lesions in mammograms. The perceptual operating points suggest superior performance for the mammographers and equivalent performance for residents in the first and second mammography rotations. These results and the model validation are preliminary as they are based on a small dataset. The significance of this study is showing that it is possible to probabilistically determine if a mark resulted from seeing a lesion or a nonlesion. Using the method developed in this study one could perform acceptance-radius independent estimation of observer performance.

show abstract

Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals—Rating-method data

Cited by 812 publications

References 6 publications

Association study between single-nucleotide polymorphisms in 199 drug-related genes and commonly measured quantitative traits of 752 healthy Japanese subjects

Association study between single-nucleotide polymorphisms in 199 drug-related genes and commonly measured quantitative traits of 752 healthy Japanese subjects

The use of the area under the ROC curve in the evaluation of machine learning algorithms

Spatial Localization Accuracy of Radiologists in Free-Response Studies: Inferring Perceptual FROC Curves from Mark-Rating Data

Contact Info

Product

Resources

About