A Thousand Words in a Scene

Abstract

This paper presents a novel approach for visual scene modeling and classification, investigating the combined use of text modeling methods and local invariant features. Our work attempts to elucidate (1) whether a text-like bag-of-visterms representation (histogram of quantized local visual features) is suitable for scene (rather than object) classification, (2) whether some analogies between discrete scene representations and text documents exist, and (3) whether unsupervised, latent space models can be used both as feature extractors for the classification task and to discover patterns of visual co-occurrence. Using several data sets, we validate our approach, presenting and discussing experiments on each of these issues. We first show, with extensive experiments on binary and multi-class scene classification tasks using a 9,500-image data set, that the bag-of-visterms representation consistently outperforms classical scene classification approaches. In other data sets we show that our approach competes with or outperforms other recent, more complex, methods. We also show that Probabilistic Latent Semantic Analysis (PLSA) generates a compact scene representation, discriminative for accurate classification, and more robust than the bag-of-visterms representation when less labeled training data is available. Finally, through aspect-based image ranking experiments, we show the ability of PLSA to automatically extract visually meaningful scene patterns, making such representation useful for browsing image collections.

Enhanced Random Forest with Image/Patch-Level Learning for Image Understanding

“…Topic models are widely applied in image classification [20]. The topic models are particularly effective when pairing with the BoW representation, where the models group ambiguous codewords together and generate a topic distribution over a codebook.…”

Mining Layered Grammar Rules for Action Recognition

“…(2) To cluster STIPs, K-means is used in the feature space of the interest points. Recently, semantic based clustering strategies are proposed to resolve the difficulties in selecting a proper K value for the K-means algorithm and the disagreement between appearance similarity and semantic consistency (Quelhas et al 2007). Based on Dollár's ST interest point detector, Niebles et al model actions using a bag-ofword model, and cluster the interest points by the underlying "topics" (Niebles et al 2008).…”

Toward a higher-level visual representation for content-based image retrieval

“…Another similar part-based image represenations that are proposed recentlty are visterms [15,23,24], SIFT-bags [39] blobs [7], and VLAD [14] vector representation of an image which aggregates descriptors based on a locality criterion in the feature space. The different approach is the one proposed by Morand et al [21].…”