On the choice of acceptance radius in free-response observer performance studies

Abstract

Objectives: Choosing an acceptance radius or proximity criterion is necessary to analyse free-response receiver operating characteristic (FROC) observer performance data. This is currently subjective, with little guidance in the literature about what is an appropriate acceptance radius. We evaluated varying acceptance radii in a nodule detection task in chest radiography and suggest guidelines for determining an acceptance radius. Methods: 80 chest radiographs were chosen, half of which contained nodules. We determined each nodule's centre. 21 radiologists read the images. We created acceptance radii bins of ,5 pixels, ,10 pixels, ,20 pixels and onwards up to ,200 and 200+ pixels. We counted lesion localisations in each bin and visually compared marks with the borders of nodules. Results: Most reader marks were tightly clustered around nodule centres, with tighter clustering for smaller than for larger nodules. At least 70% of readers' marks were placed within ,10 pixels for small nodules, ,20 pixels for medium nodules and ,30 pixels for large nodules. Of 72 inspected marks that were less than 50 pixels from the centre of a nodule, only 1 fell outside the border of a nodule. Conclusion: The acceptance radius should be based on the larger nodule sizes. For our data, an acceptance radius of 50 pixels would have captured all but 2 reader marks within the borders of a nodule, while excluding only 1 true-positive mark. The choice of an acceptance radius for FROC analysis of observer performance studies should be based on the size of larger abnormalities. Observer performance studies are often used to evaluate imaging systems in medicine. These studies are usually organised so that observers search for a particular abnormality in a set of images, indicate whether the abnormality is present and then do the same thing again on another occasion under different circumstances. There is a variety of ways in which to judge the accuracy of the observers' responses. Perhaps the most widely used method is one in which observers indicate the presence or absence of the searched-for lesion and the level of confidence with which they have identified or excluded the lesion. From this information, a receiver operating characteristic (ROC) curve is generated [1][2][3][4][5]. The free-response ROC (FROC) method is an alternative approach to analysis that uses lesion location information, and thus more closely mimicks those actual clinical practices that involve a challenging search and yields a higher statistical power than the ROC method [6,7]. Free-response methods also allow separate analysis of success in finding more than one abnormality per image [7]. In the free-response method, the observer locates each lesion, marks it and assigns a confidence rating to each marked lesion. This method is intended to avoid counting a response as correct in situations in which the observer, although correctly indicating the presence of an abnormality in an image actually containing the abnormality, was led astray by a false-positive and did not ...

Does Expectation of Abnormality Affect the Search Pattern of Radiologists When Looking for Pulmonary Nodules?

“…Viewing Images were displayed on a Viewsonic VG810b monitor (ViewSonic, Walnut, Calif) with a screen resolution of 1280 × 1024 pixels using a graphics card (NVIDIA Quadro FX 560; Nvidia, Santa Clara, California) that exceeded the minimum recommendation by the American Association of Physicists in Medicine [18]. On each day of the study, the monitor was calibrated to the Digital Imaging and Communications in Medicine grey-scale display function standard using Verilum software and luminance pod (Verilum; Image Smiths, Bethesda, MD).…”

Number of mammography cases read per year is a strong predictor of sensitivity

“…Location sensitivity is the proportion of TP marks identified in the correct location as defined by a 50-pixel radius from the center of the lesion. 30 These were calculated using the Ziltron software, and radiologists received an accuracy score at the end of the study.…”

The impact of greyscale inversion for nodule detection in an anthropomorphic chest phantom: a free-response observer study

“…The acceptance radius was determined by the largest nodules, as per current recommendations in using this methodology. 22 To establish the correct acceptance radius, nodule diameter was measured in ImageJ (Bethesda, MD). 23 Owing to magnification, the measured nodule sizes were slightly larger than the manufacturer specification.…”