Practical Issues of Experimental ROC Analysis Selection of Controls

Gur, David; King, Josh; Rockette, Howard E.; Britton, Cynthia A.; Thaete, F. Leland; Hoy, R. J.

doi:10.1097/00004424-199005000-00019

Cited by 23 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Receiver operating characteristic analysis has become a favored method for reflecting diagnostic accuracy [30]- [33], although there are acknowledged problems of spectrum bias in control patients [34], and problems of establishing the true (or gold standard) diagnosis [35]. In an ROC experiment each evaluated image or movie is assigned a rating, and by convention a higher rating indicates greater evidence of the presence of an abnormality.…”

Section: B Diagnostic-accuracy Efficacymentioning

confidence: 99%

A Methodology for Anatomic Ultrasound Image Diagnostic Quality Assessment

Hemmsen

Lange

Brandt

et al. 2017

IEEE Trans. Ultrason., Ferroelect., Freq. Contr.

View full text Add to dashboard Cite

Abstract-This paper discusses methods for assessment of ultrasound image quality based on our experiences with evaluating new methods for anatomic imaging. It presents a methodology to ensure a fair assessment between competing imaging methods using clinically relevant evaluations. The methodology is valuable in the continuing process of method optimization and guided development of new imaging methods. It includes a three phased study plan covering from initial prototype development to clinical assessment. Recommendations to the clinical assessment protocol, software, and statistical analysis are presented. Earlier uses of the methodology has shown that it ensures validity of the assessment, as it separates the influences between developer, investigator, and assessor once a research protocol has been established. This separation reduces confounding influences on the result from the developer to properly reveal the clinical value. The paper exemplifies the methodology using recent studies of Synthetic Aperture Sequential Beamforming tissue harmonic imaging.

show abstract

Section: B Diagnostic-accuracy Efficacymentioning

confidence: 99%

A Methodology for Anatomic Ultrasound Image Diagnostic Quality Assessment

Hemmsen

Lange

Brandt

et al. 2017

IEEE Trans. Ultrason., Ferroelect., Freq. Contr.

View full text Add to dashboard Cite

show abstract

“…Here we used three different probability density functions (PDF) to draw random samples: Gaussian (a common assumption in medical diagnostic performance evaluation 1,2 ), Beta (appropriate for modeling highly polarized scores like the detection of pneumothoraxes 3 ), and Uniform (providing a piecewise constant ROC curve). All distribution parameters were adjusted so that an AUC of 0.8 was achieved (see Fig.…”

Section: Simulated Datamentioning

confidence: 99%

“…In order to achieve a proper and smooth receiver operating characteristic (ROC) curve, the scoring should be evenly distributed, 1,2 which is generally hard to achieve. Furthermore, the observers may have difficulties using the rating scale consistently throughout an experiment, 3 and the observers should be familiar with the image genre to avoid inconsistent scoring within the lengthy study. 1 In this paper, we aim to reduce the complexity and duration of human reader studies using multiple rounds of a two-alternative forced choice (2AFC) paradigm while regenerating a full confidence-scoring scale.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Full receiver operating characteristic curve estimation using two alternative forced choice studies

Massanes

Brankov

2016

J. Med. Imag

View full text Add to dashboard Cite

Abstract. Task-based medical image quality is typically measured by the degree to which a human observer can perform a diagnostic task in a psychophysical human observer study. During a typical study, an observer is asked to provide a numerical score quantifying his confidence as to whether an image contains a diagnostic marker or not. Such scores are then used to measure the observers' diagnostic accuracy, summarized by the receiver operating characteristic (ROC) curve and the area under ROC curve. These types of human studies are difficult to arrange, costly, and time consuming. In addition, human observers involved in this type of study should be experts on the image genre to avoid inconsistent scoring through the lengthy study. In two-alternative forced choice (2AFC) studies, known to be faster, two images are compared simultaneously and a single indicator is given. Unfortunately, the 2AFC approach cannot lead to a full ROC curve or a set of image scores. The aim of this work is to propose a methodology in which multiple rounds of the 2AFC studies are used to re-estimate an image confidence score (a.k.a. rating, ranking) and generate the full ROC curve. In the proposed approach, we treat image confidence score as an unknown rating that needs to be estimated and 2AFC as a two-player match game. To achieve this, we use the ELO rating system, which is used for calculating the relative skill levels of players in competitor-versus-competitor games such as chess. The proposed methodology is not limited to ELO, and other rating methods such as TrueSkill™, Chessmetrics, or Glicko can be also used. The presented results, using simulated data, indicate that a full ROC curve can be recovered using several rounds of 2AFC studies and that the best pairing strategy starts with the first round of pairing abnormal versus normal images (as in the classical 2AFC approach) followed by a number of rounds using random pairing. In addition, the proposed method was tested in a pilot human observer study. These pilot results indicate that three to five rounds of 2AFC studies require less human observer time than a full scoring study and that the re-estimated ROC curves and associated area under ROC curve values have high statistical agreement with the full scoring study.

show abstract

“…instead of giving binary answer (i.e. present or absent) (Obuchowski 2003 ; Metz 1978 ; Hanley & McNeil 1982 ; Berbaum et al 1989 ; Metz 1989 ; Gur et al 1989 ). The fundamental problem of rating is that the decision needs to be “not obvious”, and “should be of borderline difficulty” (Metz 1978 ).…”

Section: Introductionmentioning

confidence: 99%

Experimental system for measurement of radiologists’ performance by visual search task

et al. 2013

View full text Add to dashboard Cite

PurposeDetective performance of radiologists for “obvious” targets should be evaluated by visual search task instead of ROC analysis, but visual task have not been applied to radiology studies. The aim of this study was to set up an environment that allows visual search task in radiology, to evaluate its feasibility, and to preliminarily investigate the effect of career on the performance.Materials and methodsIn a darkroom, ten radiologists were asked to answer the type of lesion by pressing buttons, when images without lesions, with bulla, ground-glass nodule, and solid nodule were randomly presented on a display. Differences in accuracy and reaction times depending on board certification were investigated.ResultsThe visual search task was successfully and feasibly performed. Radiologists were found to have high sensitivity, specificity, positive predictive values and negative predictive values in non-board and board groups. Reaction time was under 1 second for all target types in both groups. Board radiologists were significantly faster in answering for bulla, but there were no significant differences for other targets and values.ConclusionWe developed an experimental system that allows visual search experiment in radiology. Reaction time for detection of bulla was shortened with experience.

show abstract

Practical Issues of Experimental ROC Analysis Selection of Controls

Cited by 23 publications

References 0 publications

A Methodology for Anatomic Ultrasound Image Diagnostic Quality Assessment

A Methodology for Anatomic Ultrasound Image Diagnostic Quality Assessment

Full receiver operating characteristic curve estimation using two alternative forced choice studies

Experimental system for measurement of radiologists’ performance by visual search task

Contact Info

Product

Resources

About