Fine-Grained Representation Learning and Recognition by Exploiting Hierarchical Semantic Embedding

Abstract

Object categories inherently form a hierarchy with di erent levels of concept abstraction, especially for ne-grained categories. For example, birds (Aves) can be categorized according to a four-level hierarchy of order, family, genus, and species. This hierarchy encodes rich correlations among various categories across di erent levels, which can e ectively regularize the semantic space and thus make prediction less ambiguous. However, previous studies of negrained image recognition primarily focus on categories of one certain level and usually overlook this correlation information. In this work, we investigate simultaneously predicting categories of di erent levels in the hierarchy and integrating this structured correlation information into the deep neural network by developing a novel Hierarchical Semantic Embedding (HSE) framework. Specically, the HSE framework sequentially predicts the category score vector of each level in the hierarchy, from highest to lowest. At each level, it incorporates the predicted score vector of the higher level as prior knowledge to learn ner-grained feature representation. During training, the predicted score vector of the higher level is also employed to regularize label prediction by using it as soft targets of corresponding sub-categories. To evaluate the proposed framework, we organize the 200 bird species of the Caltech-UCSD birds dataset with the four-level category hierarchy and construct a large-scale butter y dataset that also covers four level categories. Extensive experiments on these two and the newly-released VegFru datasets demonstrate the superiority of our HSE framework over the baseline methods and existing competitors.

Self-Layer and Cross-Layer Bilinear Aggregation for Fine-Grained Recognition in Cyber-Physical-Social Systems

“…In Table 3, we can see that SCBA achieves better result compared with others weakly supervised approaches, even some supervised methods, e.g., Mask-CNN [44] and HSnet [45], which proves the validity of our model. Compared with HSE [18] which used hierarchical semantic embedding to learn stronger representation of fine-grained feature, we improve relative accuracy with 0.16% by our SCBA. We even surpass TASN [46] and DCL [47] which were the state-of-the-art weakly supervised models recently proposed, with 0.36% and 0.46% relative accuracy improves, respectively.…”

“…Generally speaking, the aims of fine-grained classification methods are high accuracy and small computational cost. In order to get better performance, Chen et al [18] exploited semantic guided attention mechanism to learn more discriminative regions at each level by incorporating the predicted score vector of the higher level as prior knowledge. Wang et al [19] proposed to add supervision information to filters for optimizing discriminative part detectors and further localizing the key regions.…”

Hierarchical bilinear convolutional neural network for image classification

“…• Coare-to-fine CNN: Fu et al [20] propose the coarseto-fine layer by applying Bayesian techniques into the network, making the coarse-to-fine network can learning the hierarchical category tree. • Tree-CNN: Roy et al [24] propose an adaptive hierarchical network structure composed of DCNNs that can grow and learn as new data becomes available.…”

Efficient Crowd Counting via Structured Knowledge Transfer

“…Recently, knowledge distillation [19], which trains a small student network to acquire the knowledge of a complex teacher network, has become a desirable alternative for model compression due to its broad applicability areas. Numerous works [9,42,46,49,69,75] Table 1: The Root Mean Squared Error (RMSE), Parameters, FLOPs, and inference time of our SKT network and four state-of-the-art models on the UCF-QNRF [22] dataset. The FLOPs and parameters are computed with the input size of 2032×2912, and the inference times are measured on a Intel Xeon E5 CPU (2.4G) and a single Nvidia GTX 1080 GPU.…”