Visual Recognition with Humans in the Loop

Abstract

We present an interactive, hybrid human-computer method for object classification. The method applies to classes of objects that are recognizable by people with appropriate expertise (e.g., animal species or airplane model), but not (in general) by people without such expertise. It can be seen as a visual version of the 20 questions game, where questions based on simple visual attributes are posed interactively. The goal is to identify the true class while minimizing the number of questions asked, using the visual content of the image. We introduce a general framework for incorporating almost any off-the-shelf multi-class object recognition algorithm into the visual 20 questions game, and provide methodologies to account for imperfect user responses and unreliable computer vision algorithms. We evaluate our methods on Birds-200, a difficult dataset of 200 tightly-related bird species, and on the Animals With Attributes dataset. Our results demonstrate that incorporating user input drives up recognition accuracy to levels that are good enough for practical applications, while at the same time, computer vision reduces the amount of human interaction required.

Fine-Grained Visual Comparisons with Local Learning

“…Work on fine-grained visual categorization aims to recognize objects in a single domain, e.g., bird species [8,14]. While such problems also require making distinctions among visually close instances, our goal is to compare attributes, not categorize objects.…”

Attribute Pivots for Guiding Relevance Feedback in Image Search

“…2 We generate the response for, "Is the target image more, equally, or less m than I pm ?" using the difference in the predicted attribute values for the target and I pm .…”

“…[6,2]. In the case where a human answers the tests, attributes are well-suited to query for intermediate labels that will lead to the right category label, as shown for bird labeling [2]. Our work shares the spirit of rapidly reducing uncertainty through a sequence of useful questions.…”

“…Active testing methods choose a series of useful "tests" (e.g., features to extract) or label requests ("does the bird have a yellow beak?") [6,2]. In the case where a human answers the tests, attributes are well-suited to query for intermediate labels that will lead to the right category label, as shown for bird labeling [2].…”

“…The basic idea of expected error reduction was first proposed in [18] for active learning in text classification, and variations have been explored in vision tasks (e.g., [2,9,14]). Our formulation is novel in that we survey only the attribute pivots, exploiting the special structure of rankable visual properties for substantial computational savings.…”