The Bayesian image retrieval system, PicHunter: theory, implementation, and psychophysical experiments

Abstract

This paper presents the theory, design principles, implementation and performance results of PicHunter, a prototype content-based image retrieval (CBIR) system. In addition, this document presents the rationale, design and results of psychophysical experiments that were conducted to address some key issues that arose during PicHunter's development. The PicHunter project makes four primary contributions to research on CBIR. First, PicHunter represents a simple instance of a general Bayesian framework which we describe for using relevance feedback to direct a search. With an explicit model of what users would do, given the target image they want, PicHunter uses Bayes's rule to predict the target they want, given their actions. This is done via a probability distribution over possible image targets, rather than by refining a query. Second, an entropy-minimizing display algorithm is described that attempts to maximize the information obtained from a user at each iteration of the search. Third, PicHunter makes use of hidden annotation rather than a possibly inaccurate/inconsistent annotation structure that the user must learn and make queries in. Finally, PicHunter introduces two experimental paradigms to quantitatively evaluate the performance of the system, and psychophysical experiments are presented that support the theoretical claims.

Attribute Pivots for Guiding Relevance Feedback in Image Search

“…The goal is to solicit feedback on those exemplars that would most improve the system's notion of relevance. Many existing methods exploit classifier uncertainty to find useful exemplars (e.g., [23,11,3,26]). However, traditional approaches have two main limitations.…”

“…The benefits of interactive feedback for image search are well studied [3,19,26,5]. In practice, the images displayed to the user for feedback are usually those ranked best by the system's current relevance model.…”

“…Notably, prior efforts to display the exemplar set that minimizes uncertainty were forced to resort to sampling or clustering heuristics due to the combinatorial optimization problem inherent when categorical feedback is assumed (e.g., [18,3,5]). In contrast, we show that eliciting comparative feedback on ordinal visual attributes naturally leads to an efficient sequential selection strategy, where each comparison is guaranteed to decrease the predicted relevance of half of the unexplored database images.…”

“…Our formulation is novel in that we survey only the attribute pivots, exploiting the special structure of rankable visual properties for substantial computational savings. In contrast, existing work resorts to sampling heuristics [3], approximations [5], or simply small data pools [9] to make the problem tractable.…”

“…Due to the well known semantic gap, one-shot retrieval is generally insufficient. Instead, an interactive approach lets the user help the system refine the top-ranked results via iterative feedback [3,19,13,23,11,26,5]. The most common form of interaction consists of binary relevance feedback, in which the user declares certain exemplars to be relevant or irrelevant, and then the system updates its relevance metric in response.…”