<title>Benchmark for image retrieval using distributed systems over the Iinternet: BIRDS-I</title>

Abstract

Comparing the performance of CBIR (Content-Based Image Retrieval) algorithms is difficult. Private data sets are used so it is controversial to compare CBIR algorithms developed by different researchers. Also, the performance of CBIR algorithms is usually measured on an isolated, well-tuned PC or workstation. In a real-world environment, however, the CBIR algorithms would only constitute a minor component among the many interacting components needed to facilitate a useful CBIR application e.g., Web-based applications on the Internet. The Internet, being a shared medium, dramatically changes many of the usual assumptions about measuring CBIR performance. Any CBIR benchmark should be designed from a networked systems standpoint. Networked system benchmarks have been developed for other applications e.g., text retrieval, and relational database management. These benchmarks typically introduce communication overhead because the real systems they model are distributed applications e.g., an airline reservation system. The most common type of distributed computing architecture uses a client/server model. We present our implementation of a client/server CBIR benchmark called BIRDS-I (Benchmark for Image Retrieval using Distributed Systems over the Internet) to measure image retrieval performance over the Internet. The BIRDS-I benchmark has been designed with the trend toward the use of small personalized wireless-internet systems in mind. Web-based CBIR implies the use of heterogeneous image sets and this, in turn, imposes certain constraints on how the images are organized and the type of performance metrics that are applicable. Surprisingly, BIRDS-I only requires controlled human intervention for the compilation of the image collection and none for the generation of ground truth in the measurement of retrieval accuracy. Benchmark image collections need to be evolved incrementally toward the storage of millions of images and that scaleup can only be achieved through the use of computeraided compilation. Finally, the BIRDS-I scoring metric introduces a tightly optimized image-ranking window, which is important for the future benchmarking of large-scale personalized wireless-internet CBIR systems.

Prosemantic Features for Content-Based Image Retrieval

“…The dataset used consists of 1875 images taken from the Benchathlon dataset [31]. The dataset includes typical consumer photographs showing a very different distribution of concepts with respect to the dataset used to train the classifiers.…”

Evaluation strategies for image understanding and retrieval

“…More automatic methods typically involve having sets of images tagged with high level concepts (e.g., sky, grass ), and retrieval is evaluated based on those labels [45,46,47], making performance evaluation similar to that in text retrieval [39]. The Benchatholon project proposes providing much more detailed and publicly available keywords of images using a controlled vocabulary [23,37,1]. A problem with both these approaches is that they are only indirectly connected to the task that they are trying to measure.…”

Content-Based Image Retrieval Using Self-Organizing Maps

“…Evaluation benchmarks should therefore be designed to use sufficiently large data sets. For example, in Gunther and Beretta (2000), it was suggested that even an initial benchmark database should contain at least 10 000 images. The scalability requirement has also been long recognized in the IR community and TREC, for example, has always concentrated on retrieval from large test collections.…”

“…The aim of the initiative is to set up a collaborative environment where standard CBIR evaluation protocols and frameworks can be developed. The leading principle in designing the benchmark has been to use a distributed client-server architecture, as described in Gunther and Beretta (2000). The purpose is to divide CBIR systems into separate client and server parts, in order to be able to measure CBIR performance over the Internet.…”

“…Variations of the average rank include normalized average rank (Müller et al 2001), the BIRDS-I measure (Gunther and Beretta 2000) and the so-called τ measure used by our research group (Publications I, III, IV, and VI). The τ measure coincides with the question "how large portion of the whole database needs to be browsed through until, on the average, the searched image will be found" when we assume that one (random) image from the class in question is the actual target image the user is looking for and she uses categorical feedback to guide the retrieval system.…”