The SUN Attribute Database: Beyond Categories for Deeper Scene Understanding

Abstract

In this paper we present the first large-scale scene attribute database. First, we perform crowdsourced human studies to find a taxonomy of 102 discriminative attributes. We discover attributes related to materials, surface properties, lighting, affordances, and spatial layout. Next, we build the "SUN attribute database" on top of the diverse SUN categorical database. We use crowdsourcing to annotate attributes for 14,340 images from 707 scene categories. We perform numerous experiments to study the interplay between scene attributes and scene categories. We train and evaluate attribute classifiers and then study the feasibility of attributes as an intermediate scene representation for scene classification, zero shot learning, automatic image captioning, semantic image search, and parsing natural images. We show that when used as features for these tasks, low dimensional scene attributes can compete with or improve on the state of the art performance. The experiments suggest that scene attributes are an effective low-dimensional feature for capturing high-level context and semantics in scenes.

Ambiguity Helps: Classification with Disagreements in Crowdsourced Annotations

“…We use a publicly available SUN Attribute dataset (SUNAttribute) 4 [20] that comes with the averaged score over MTurk annotations of attribute being present in the image. In the second experiment, we focus on differentiating between 'easy' and 'hard' images of animal classes.…”

“…Fast forward to 2008 and beyond, the needs for learning with multiple noisy annotations are further exemplified by the advent of crowdsourcing platforms [30,24,37,1,2,15]. Prior work ranges from a simple majority voting where all annotators are weighted equally [20,26] to a weighted voting by quantifying the expertise of the annotators [8]. Work that actively selects both the informative instances and the high-quality annotators also exists [25,15].…”

“…With MTurk, now it becomes possible to collect annotations for large datasets such as ImageNet [26], TinyImages [31], COCO [14], and Places [38]. Moreover, it becomes prevalent to collect task-specific datasets, for example for studying the attributes and their strength [20] and for determining the easiness or hardness of a particular classification task [22]. Those task-specific datasets often require annotations that are more ambiguous than typical object annotations 'present' or 'not present'.…”

“…A confidence in the label annotations, ranging from a simple percentage agreement to the kappa statistic, is then computed from the multiple annotations at each data point. For sufficiently high confidence annotations, a majority voting scheme is then widely adopted as the ground-truth label [31,14,38,20,22,26].…”

Exploring the Long Tail of Social Media Tags

“…Many object tags are available in ImageNet [2], 22K classes resulting in 40K tags, since some classes come with multiple tags, like sea cow, sirenian mammal, sirenian represent one object class. The SUN Attribute dataset [15] has the highest number of scene tags so far, 717 classes like amusement park, coast, squash court. Tags from fine-grained animal categories are available in the 120 dog tag from Stanford Dogs [5] like pekinese, irish terrier, chihuahua, and 200 bird tags from Caltech Birds [22] cowbird, bobolink, blue jay.…”