Object-aware Long-short-range Spatial Alignment for Few-Shot Fine-Grained Image Classification

Abstract

The goal of few-shot fine-grained image classification is to recognize rarely seen fine-grained objects in the query set, given only a few samples of this class in the support set. Previous works focus on learning discriminative image features from a limited number of training samples for distinguishing various fine-grained classes, but ignore one important fact that spatial alignment of the discriminative semantic features between the query image with arbitrary changes and the support image, is also critical for computing the semantic similarity between each support-query pair. In this work, we propose an object-aware long-short-range spatial alignment approach, which is composed of a foreground object feature enhancement (FOE) module, a long-range semantic correspondence (LSC) module and a short-range spatial manipulation (SSM) module. The FOE is developed to weaken background disturbance and encourage higher foreground object response. To address the problem of long-range object feature misalignment between support-query image pairs, the LSC is proposed to learn the transferable long-range semantic correspondence by a designed feature similarity metric. Further, the SSM module is developed to refine the transformed support feature after the long-range step to align short-range misaligned features (or local details) with the query features. Extensive experiments have been conducted on four benchmark datasets, and the results show superior performance over most state-of-the-art methods under both 1-shot and 5-shot classification scenarios. CCS CONCEPTS• Computing methodologies → Visual content-based indexing and retrieval; Matching.

Learning Cross-Image Object Semantic Relation in Transformer for Few-Shot Fine-Grained Image Classification

“…In this stage, the backbone is trained from scratch using SGD optimizer with a batch size of 128, a momentum of 0.9, a weight decay of 0.0005, and an initial learning rate of 0.1. To keep consistent with the setting in [58], the learning rate decays at 85 and 170 epochs. We remove the fully-connected layer for performing the next meta-training stage.…”

“…Backbone CUB → NABirds 1-shot 5-shot LSC+SSM (Baseline) [58] ResNet-12 45.70±0. 45 4: 5-way few-shot fine-grained classification results by adapting from the CUB-trained model to NABirds dataset using different backbones.…”

“…Besides, the work [57] tackles the FSFGL problem from the perspective of reconstructing the query image to learn a classifier. More recently, the work [58] tries to increase the fine-grained classification accuracy via long-shot-range spatial alignment between support and query features. Motivated by these works in the FSFGL community, we further extend the study of FSFGL to a Transformer-based structure, and investigate its effectiveness in strengthening the support-query relation matching process only given a few samples.…”

“…Therefore, researchers mainly focus on leveraging meta-learning technology [13,39,41,43,48,59] to deal with the FSL problem. However, the above-mentioned FSL methods focus to classify coarse-grained generic object categories, which are less suitable to address the few-shot fine-grained classification task [29,57,58,71], that requires to emphasize the local feature variations or subtle feature differences.…”

“…As illustrated in Fig. 1(a), many few-shot fine-grained works mine discriminative parts of the whole image based on the attention mechanism [71], feature map reconstruction [57], and feature-level spatial alignment [58]. However, these methods fail to As a matter of fact, to recognize a novel class's samples, humans tend to compare the local semantic parts' differences between the ever-seen and newly-seen objects, so that the subtle feature differences can be identified using only a few examples.…”