Evaluating the Performance of Content-Based Image Retrieval Systems

VISTO: A new CBIR system for vector images

“…As frequently pointed out by researchers in this field (e.g., see [21,61]), the image retrieval research community is using a wide variety of measures for evaluating image retrieval performance, in contrast to the text retrieval research community, which has long been using standard consolidated quality indicators, namely, the well known Precision and Recall measures (see, e.g., [19,25,26]). As pointed out for example by Smith in [61], there are many benefits in adopting Precision and Recall as the basis for image retrieval evaluation as well, not only because they are standard measures in information retrieval, but also because many of the existing measures for image retrieval can be computed by them.…”

“…While researchers in text image retrieval have long been using a sophisticated set of tools for user-based evaluation, this does not yet apply to image retrieval [21]. In order to evaluate the effectiveness and the efficiency of VISTO, we use the two well known measures, Precision and Recall (see, e.g., [19,25,26,61]), and study the behavior of Precision at different values of Recall, that is the so-called Precision versus Recall curves. The effectiveness of our system is demonstrated by the fact that the behavior of these curves is always descendent.…”

A web-based evaluation system for CBIR

“…This measure is 0 for perfect , 0.5 for random performance, and approaches 1 as performance worsens. We use several measures based on precision and recall despite criticism on precision and recall already in the 60s [16] and as well in [3,7] for the use in CBIR. Precision and recall, especially in form of the PR graph and in form of precision or recall at important cutoff point, are still the standard in TR and are easy to interpret.…”

“…It is also important not to compare the systems based on a single performance measure, but on several measures because for different application areas different characteristics are important. Koskela et al [7] describe performance measures to quantify how close together clusters of images are in feature space based on their ranks. This can be used to compare different features and techniques.…”

Complementary cumulative precision distribution: a new graphical metric for medical image retrieval system

“…There exist no single widely accepted performance assessment method that can be used for objective and quantitative comparison between different CBIR systems based on different approaches. 2 Several single valued measures have been proposed by researchers. Precision and recall are the most common evaluation measure used by researchers.…”