Con-Text: Text Detection for Fine-Grained Object Classification

Abstract

This paper focuses on fine-grained object classification using recognized scene text in natural images. While the state-of-the-art relies on visual cues only, this paper is the first work which proposes to combine textual and visual cues. Another novelty is the textual cue extraction. Unlike the state-of-the-art text detection methods, we focus more on the background instead of text regions. Once text regions are detected, they are further processed by two methods to perform text recognition, i.e., ABBYY commercial OCR engine and a state-of-the-art character recognition algorithm. Then, to perform textual cue encoding, bi- and trigrams are formed between the recognized characters by considering the proposed spatial pairwise constraints. Finally, extracted visual and textual cues are combined for fine-grained classification. The proposed method is validated on four publicly available data sets: ICDAR03, ICDAR13, Con-Text, and Flickr-logo. We improve the state-of-the-art end-to-end character recognition by a large margin of 15% on ICDAR03. We show that textual cues are useful in addition to visual cues for fine-grained classification. We show that textual cues are also useful for logo retrieval. Adding textual cues outperforms visual- and textual-only in fine-grained classification (70.7% to 60.3%) and logo retrieval (57.4% to 54.8%).

Words Matter: Scene Text for Image Classification and Retrieval

“…Additional pairwise spatial constraints between characters are used to refine the ranking. Karaoglu et al [24] propose to use textual cues in combination with visual cues for fine-grained classification. Bi-grams are computed based on recognized characters in images.…”

“…Fine-grained classification is the problem of the categorization of subordinate-level categories such as bird species [57], flower types [39] and business places [24]. The small interclass visual differences and the large intra-class variations make fine-grained classification challenging.…”

“…We use the Con-Text dataset proposed in [24]. The dataset is for fine-grained classification of business places, e.g., Cafe, Bookstore and Pharmacy.…”

“…First, as in [24], we use bag of visual words representation with 3×1 and 2×2 spatial pyramid, and histogram intersection kernel. This baseline is denoted as BOW.…”

“…We compare our approach with the stateof-the-art [24] which extracts the textual information at a char- [24]. Where the visual-only performance is compatible, the proposed method outperforms [24] by a large margin. It can be derived that representing the textual information at word level is more effective than at a character level.…”