Semantic Segmentation In-the-Wild Without Seeing Any Segmentation Examples

Abstract

Semantic segmentation is a key computer vision task that has been actively researched for decades. In recent years, supervised methods have reached unprecedented accuracy, however they require many pixel-level annotations for every new class category which is very time-consuming and expensive. Additionally, the ability of current semantic segmentation networks to handle a large number of categories is limited. That means that images containing rare class categories are unlikely to be well segmented by current methods. In this paper we propose a novel approach for creating semantic segmentation masks for every object, without the need for training segmentation networks or seeing any segmentation masks. Our method takes as input the image-level labels of the class categories present in the image; they can be obtained automatically or manually. We utilize a vision-language embedding model (specifically CLIP) to create a rough segmentation map for each class, using model interpretability methods. We refine the maps using a test-time augmentation technique. The output of this stage provides pixel-level pseudo-labels, instead of the manual pixel-level labels required by supervised methods. Given the pseudo-labels, we utilize single-image segmentation techniques to obtain high-quality output segmentation masks. Our method is shown quantitatively and qualitatively to outperform methods that use a similar amount of supervision. Our results are particularly remarkable for images containing rare categories.

GroupViT: Semantic Segmentation Emerges from Text Supervision

“…The concurrently developed unpublished text-supervised semantic segmentation methods [29,86,90,96] also show promising results. One major difference between these methods and GroupViT is that, they exploit vision-language model [32,61] pre-trained on well prepared large-scale 400M-1.8B image-text data, while our GroupViT is trained from scratch with much noisier data (30M images) to learn grouping and segmentation and yet achieves competitive performance.…”

Causal Attention for Vision-Language Tasks

“…Propose language-driven semantic segmentation by matching pixel and text embeddings. SSIW [187] Introduce a test-time augmentation technique to refine the pseudo labels generated by CLIP. MaskClip+ [180] [code]…”

“…In addition, ZegCLIP [188] employs CLIP to generate semantic masks and introduces a relationship descriptor to mitigate overfitting on base classes in knowledge distillation. MaskCLIP+ [180] and SSIW [187] distill knowledge with VLM-predicted pixel-level pseudo labels, where SSIW improves the pseudo labels by test-time augmentation. Knowledge distillation for weakly-supervised semantic segmentation aims to leverage both VLMs and weak supervision (e.g., image-level labels) for semantic segmentation.…”