Mining Latent Classes for Few-shot Segmentation

Yang, Lihe; Zhuo, Wei; Qi, Lei; Shi, Yinghuan; Gao, Yang

doi:10.1109/iccv48922.2021.00860

Cited by 84 publications

(34 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Multiple research directions have been proposed to learn to segment with less supervision than dense per-pixel labels. For example, few-shot learning [22,46,52,57,72,79,87] and active learning [9,65,68,69,85] are proposed to perform segmentation with as few pixel-wise labels as possible. Going further, zero-shot approaches [7,40] are proposed to learn segmentation models for unseen categories without using pixel-wise labels for them.…”

Section: Related Workmentioning

confidence: 99%

GroupViT: Semantic Segmentation Emerges from Text Supervision

Xu¹,

Mello²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Grouping and recognition are important components of visual scene understanding, e.g., for object detection and semantic segmentation. With end-to-end deep learning systems, grouping of image regions usually happens implicitly via top-down supervision from pixel-level recognition labels. Instead, in this paper, we propose to bring back the grouping mechanism into deep networks, which allows semantic segments to emerge automatically with only text supervision. We propose a hierarchical Grouping Vision Transformer (GroupViT), which goes beyond the regular grid structure representation and learns to group image regions into progressively larger arbitraryshaped segments. We train GroupViT jointly with a text encoder on a large-scale image-text dataset via contrastive losses. With only text supervision and without any pixellevel annotations, GroupViT learns to group together semantic regions and successfully transfers to the task of semantic segmentation in a zero-shot manner, i.e., without any further fine-tuning. It achieves a zero-shot accuracy of 51.2% mIoU on the PASCAL VOC 2012 and 22.3% mIoU on PASCAL Context datasets, and performs competitively to state-of-the-art transfer-learning methods requiring greater levels of supervision. Project page is available at https://jerryxu.net/GroupViT.

show abstract

Section: Related Workmentioning

confidence: 99%

GroupViT: Semantic Segmentation Emerges from Text Supervision

Xu¹,

Mello²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…PFENet [28] generated prior masks from high-level features to guide predictions. To utilize backround region of training images, [37] proposed latent class mining strategy and rectification method for support prototypes. CyCTR [41] proposed cycle-consistent mechanism and integrated it into Transformer architecture, aiming to use the information of whole support features.…”

Section: Few-shot Segmentationmentioning

confidence: 99%

“…Few-shot segmentation [37,43,40,28] aims to segment novel objects in a query image using few annotated support images. Existing methods mostly leverage representations from annotated support images and the similarity between the representations and support features for dense predictions.…”

Section: Introductionmentioning

confidence: 99%

Contrastive Enhancement Using Latent Prototype for Few-Shot Segmentation

Zhao¹,

Chen²,

Gong³

et al. 2022

Preprint

View full text Add to dashboard Cite

Few-shot segmentation enables the model to recognize unseen classes with few annotated examples. Most existing methods adopt prototype learning architecture, where support prototype vectors are expanded and concatenated with query features to perform conditional segmentation. However, such framework potentially focuses more on query features while may neglect the similarity between support and query features. This paper proposes a contrastive enhancement approach using latent prototypes to leverage latent classes and raise the utilization of similarity information between prototype and query features. Specifically, a latent prototype sampling module is proposed to generate pseudo-mask and novel prototypes based on features similarity. The module conveniently conducts end-to-end learning and has no strong dependence on clustering numbers like cluster-based method. Besides, a contrastive enhancement module is developed to drive models to provide different predictions with the same query features. Our method can be used as an auxiliary module to flexibly integrate into other baselines for a better segmentation performance. Extensive experiments show our approach remarkably improves the performance of state-of-the-art methods for 1-shot and 5-shot segmentation, especially outperforming baseline by 5.9% and 7.3% for 5-shot task on Pascal-5 i and COCO-20 i . Source code is available at https://github.com/zhaoxiaoyu1995/CELP-Pytorch

show abstract

“…The recent work on FSS focuses on modeling one-toone correspondence between support and query pixels to create correlation scores [27,44,53]. Another line of research is learning class prototypes to be used in deciding whether each pixel belongs to the object or the background [49,22,52,42,54,45,41]. To build models that are trained to generalize over few training samples, recent work widely adopts the meta learning paradigm [27,44,53,49,22,52,42,54,45] .…”

Section: Introductionmentioning

confidence: 99%

“…Another line of research is learning class prototypes to be used in deciding whether each pixel belongs to the object or the background [49,22,52,42,54,45,41]. To build models that are trained to generalize over few training samples, recent work widely adopts the meta learning paradigm [27,44,53,49,22,52,42,54,45] . In meta learning, the main idea is to create a series of tasks based on the training set simulating the few-shot segmentation problem and train the meta model over these tasks.…”

Section: Introductionmentioning

confidence: 99%

MaskSplit: Self-supervised Meta-learning for Few-shot Semantic Segmentation

Amac¹,

Sencan²,

Baran³

et al. 2021

Preprint

View full text Add to dashboard Cite

Just like other few-shot learning problems, few-shot segmentation aims to minimize the need for manual annotation, which is particularly costly in segmentation tasks. Even though the few-shot setting reduces this cost for novel test classes, there is still a need to annotate the training data. To alleviate this need, we propose a self-supervised training approach for learning few-shot segmentation models. We first use unsupervised saliency estimation to obtain pseudomasks on images. We then train a simple prototype based model over different splits of pseudo masks and augmentations of images. Our extensive experiments show that the proposed approach achieves promising results, highlighting the potential of self-supervised training. To the best of our knowledge this is the first work that addresses unsupervised few-shot segmentation problem on natural images.

show abstract

Mining Latent Classes for Few-shot Segmentation

Cited by 84 publications

References 35 publications

GroupViT: Semantic Segmentation Emerges from Text Supervision

GroupViT: Semantic Segmentation Emerges from Text Supervision

Contrastive Enhancement Using Latent Prototype for Few-Shot Segmentation

MaskSplit: Self-supervised Meta-learning for Few-shot Semantic Segmentation

Contact Info

Product

Resources

About