Collect and Select: Semantic Alignment Metric Learning for Few-Shot Learning

Hao, Fang; He, Fenghua; Cheng, Jun; Wang, Lei; Cao, Jianzhong; Tao, Dacheng

doi:10.1109/iccv.2019.00855

Cited by 114 publications

(62 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Patch-based methods first split each image into local patches and then consider the similarity between them by computing their cross-correlation and combination. Generally, the alignment task can be divided into two subtasks: the attentional alignment task [170,171] and the semantic alignment task [172,173]. The attentional alignment task is based on the attentional mapping between the features of the input modality and the target one, while the semantic alignment task takes the form of an alignment method that directly provides alignment capabilities to a predictive model.…”

Section: Multimodal Alignmentmentioning

confidence: 99%

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

et al. 2021

View full text Add to dashboard Cite

The research progress in multimodal learning has grown rapidly over the last decade in several areas, especially in computer vision. The growing potential of multimodal data streams and deep learning algorithms has contributed to the increasing universality of deep multimodal learning. This involves the development of models capable of processing and analyzing the multimodal information uniformly. Unstructured real-world data can inherently take many forms, also known as modalities, often including visual and textual content. Extracting relevant patterns from this kind of data is still a motivating goal for researchers in deep learning. In this paper, we seek to improve the understanding of key concepts and algorithms of deep multimodal learning for the computer vision community by exploring how to generate deep models that consider the integration and combination of heterogeneous visual cues across sensory modalities. In particular, we summarize six perspectives from the current literature on deep multimodal learning, namely: multimodal data representation, multimodal fusion (i.e., both traditional and deep learning-based schemes), multitask learning, multimodal alignment, multimodal transfer learning, and zero-shot learning. We also survey current multimodal applications and present a collection of benchmark datasets for solving problems in various vision domains. Finally, we highlight the limitations and challenges of deep multimodal learning and provide insights and directions for future research.

show abstract

Section: Multimodal Alignmentmentioning

confidence: 99%

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

et al. 2021

View full text Add to dashboard Cite

show abstract

“…The aforementioned methods all rely on global image features. A few methods have also been proposed that aim to identify finer-grained local features, such as DN4 [24], SAML [10], STANet [54] and CTM [23].…”

Section: Related Workmentioning

confidence: 99%

“…The results of the baselines in Table 7 (miniImageNet) are obtained from [22], [55], [38], [14] and [56]. The results for the baselines in Table 8 (CUB) are obtained from [10], [24] and [53]. These results are based on the Conv-64 and ResNet-12 backbone, which we therefore adopt as well for this dataset.…”

Section: 32mentioning

confidence: 99%

Aligning Visual Prototypes with BERT Embeddings for Few-Shot Learning

Yan

Bouraoui

Wang

et al. 2021

Proceedings of the 2021 International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

Few-shot learning (FSL) is the task of learning to recognize previously unseen categories of images from a small number of training examples. This is a challenging task, as the available examples may not be enough to unambiguously determine which visual features are most characteristic of the considered categories. To alleviate this issue, we propose a method that additionally takes into account the names of the image classes. While the use of class names has already been explored in previous work, our approach differs in two key aspects. First, while previous work has aimed to directly predict visual prototypes from word embeddings, we found that better results can be obtained by treating visual and text-based prototypes separately. Second, we propose a simple strategy for learning class name embeddings using the BERT language model, which we found to substantially outperform the GloVe vectors that were used in previous work. We furthermore propose a strategy for dealing with the high dimensionality of these vectors, inspired by models for aligning cross-lingual word embeddings. We provide experiments on miniImageNet, CUB and tieredImageNet, showing that our approach consistently improves the state-of-the-art in metric-based FSL.

show abstract

“…Different from Relation Network, semantic alignment metric learning (SAML) [69] adopted the multi-layer perceptron (MLP) network for computing the similarity score. Specifically, SAML contains a feature embedding module and a semantic alignment module.…”

Section: Learning Similarity Scores Via Neural Networkmentioning

confidence: 99%

Revisiting metric learning for few-shot image classification

et al. 2020

Neurocomputing

View full text Add to dashboard Cite

The goal of few-shot learning is to recognize new visual concepts with just a few amount of labeled samples in each class. Recent effective metric-based few-shot approaches employ neural networks to learn a feature similarity comparison between query and support examples. However, the importance of feature embedding, i.e., exploring the relationship among training samples, is neglected. In this work, we present a simple yet powerful baseline for few-shot classification by emphasizing the importance of feature embedding. Specifically, we revisit the classical triplet network from deep metric learning, and extend it into a deep K-tuplet network for few-shot learning, utilizing the relationship among the input samples to learn a general representation learning via episode-training. Once trained, our network is able to extract discriminative features for unseen novel categories and can be seamlessly incorporated with a non-linear distance metric function to facilitate the few-shot classification. Our result on the miniImageNet benchmark outperforms other metric-based few-shot classification methods. More importantly, when evaluated on completely different datasets (Caltech-101, CUB-200, Stanford Dogs and Cars) using the model trained with miniImageNet, our method significantly outperforms prior methods, demonstrating its superior capability to generalize to unseen classes.

show abstract

Collect and Select: Semantic Alignment Metric Learning for Few-Shot Learning

Cited by 114 publications

References 14 publications

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

Aligning Visual Prototypes with BERT Embeddings for Few-Shot Learning

Revisiting metric learning for few-shot image classification

Contact Info

Product

Resources

About