Evaluating Color Descriptors for Object and Scene Recognition

Abstract

Image category recognition is important to access visual information on the level of objects and scene types. So far, intensity-based descriptors have been widely used for feature extraction at salient points. To increase illumination invariance and discriminative power, color descriptors have been proposed. Because many different descriptors exist, a structured overview is required of color invariant descriptors in the context of image category recognition. Therefore, this paper studies the invariance properties and the distinctiveness of color descriptors (software to compute the color descriptors from this paper is available from http://www.colordescriptors.com) in a structured way. The analytical invariance properties of color descriptors are explored, using a taxonomy based on invariance properties with respect to photometric transformations, and tested experimentally using a data set with known illumination conditions. In addition, the distinctiveness of color descriptors is assessed experimentally using two benchmarks, one from the image domain and one from the video domain. From the theoretical and experimental results, it can be derived that invariance to light intensity changes and light color changes affects category recognition. The results further reveal that, for light intensity shifts, the usefulness of invariance is category-specific. Overall, when choosing a single descriptor and no prior knowledge about the data set and object and scene categories is available, the OpponentSIFT is recommended. Furthermore, a combined set of color descriptors outperforms intensity-based SIFT and improves category recognition by 8 percent on the PASCAL VOC 2007 and by 7 percent on the Mediamill Challenge.

Coloring Action Recognition in Still Images

“…The use of color for object recognition has been extensively studied (van de Weijer and Schmid, 2006;van de Sande et al, 2010;Bosch et al, 2008;Everingham et al, 2009;Khan et al, 2012bKhan et al, , 2011. A variety of color descriptors and approaches to combining color and shape cues for object recognition have been proposed in the literature Schmid, 2007, 2006;van de Sande et al, 2010;Bosch et al, 2006;Vigo et al, 2010).…”

“…A variety of color descriptors and approaches to combining color and shape cues for object recognition have been proposed in the literature Schmid, 2007, 2006;van de Sande et al, 2010;Bosch et al, 2006;Vigo et al, 2010). Bosch et al (2006) propose to compute SIFT descriptors directly on HSV channels of color images.…”

“…A low dimensional color descriptor based on color names was proposed by van de Weijer et al (2009). See (van de Sande et al, 2010) for a comprehensive study and evaluation of a large number of color descriptors.…”

“…Channel-based fusion for color and shape was first proposed by Bosch et al (2008) and later extensively investigated and tested by van de Sande et al (2010). First, 4 The combined vocabulary sc is constructed by concatenating the shape and color features before constructing the vocabulary in the combined feature-space.…”

“…Such variations complicate the problem of robust color description as can be seen in Figure 1. Color has, however, led to significantly improved results on other recognition tasks, such as image classification and object detection (van de Weijer and Schmid, 2006;van de Sande et al, 2010;Bosch et al, 2008;Gehler and Nowozin, 2009;Khan et al, 2012bKhan et al, , 2011. Here we investigate both color features and fusion methods to optimally incorporate color into the human action classification pipeline.…”