Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals

Gansbeke, Wouter Van; Vandenhende, Simon; Georgoulis, Stamatios; Gool, Luc Van

doi:10.48550/arxiv.2102.06191

Cited by 18 publications

(29 citation statements)

References 68 publications

(117 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Image segmentation was conducted to simplify the features of the cirrus clouds. A 128 2 x 3 (16,384 x 3) dimensional array, consisting of each pixel value of an image along with the pixel's coordinates on the image (ranging from 0 to 128), was fed into the K-Means Clustering algorithm with a K-value of 3.…”

Section: K-means Clustering For Unsupervised Cirrus Image Segmentationmentioning

confidence: 99%

Analyzing Multispectral Satellite Imagery of South American Wildfires Using Deep Learning

Sun

2022

Preprint

View full text Add to dashboard Cite

Since severe droughts are occurring more frequently and lengthening the dry season in the Amazon Rainforest, it is important to respond to active wildfires promptly and to forecast them before they become inextinguishable. Though computer vision researchers have applied algorithms on large databases to automatically detect wildfires, current models are computationally expensive and are not versatile enough for the low technology conditions of regions in South America. This comprehensive deep learning study first trains a Fully Convolutional Neural Network with skip connections on multispectral Landsat 8 images of Ecuador and the Galapagos. The model uses Green and Short-wave Infrared bands as inputs to predict each image's corresponding pixel-level binary fire mask. This model achieves a 0.962 validation F2 score and a 0.932 F2 score on test data from Guyana and Suriname. Afterward, image segmentation is conducted on the Cirrus Cloud band using K-Means Clustering to simplify continuous pixel values into three discrete classes representing the degree of cirrus cloud contamination. Two additional Convolutional Neural Networks are trained to classify the presence of a wildfire in a patch of land using these segmented cirrus images. The "experimental" model trained on the segmented inputs achieves 96.5% binary accuracy and has smoother learning curves than the "control model" that is not given the segmented inputs. This proof of concept reveals that feature simplification can improve the performance of wildfire detection models. Overall, the software built in this study is useful for early and accurate detection of wildfires in South America.

show abstract

Section: K-means Clustering For Unsupervised Cirrus Image Segmentationmentioning

confidence: 99%

Analyzing Multispectral Satellite Imagery of South American Wildfires Using Deep Learning

Sun

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Early work by [10] enforce patch-level local features across different augmented views of an image to be similar to each other while being dissimilar to representations of other local regions within the image. More recent local contrastive learning methods extend this work by using surrogate semantic labels [17], [18] where the labels are estimated on unlabeled images using supervised techniques such as saliency maps estimation approaches [19], [20], [21], super-pixels [22], and image computable masks [23], [24].. These methods encourage local regions having the same semantic label to have similar representations while being dissimilar to regions with different labels.…”

Section: A Motivationmentioning

confidence: 99%

“…Some methods [10], [44], [45], [46] pre-train by matching patch/pixel-level representations across either across two augmented views of an image or positive pair of images defined using domain cues. Other methods [18], [17] obtain some surrogate masks of foreground objects using unsupervised techniques such as saliency maps [19], [20], [21], super-pixels [22], image computable masks [23], [24]. Later, the local level features of a chosen fore-ground object are optimized to be similar while other objects local features are optimized to be dissimilar.…”

Section: Related Workmentioning

confidence: 99%

Local contrastive loss with pseudo-label based self-training for semi-supervised medical image segmentation

Chaitanya¹,

Erdil²,

Karani³

et al. 2021

Preprint

View full text Add to dashboard Cite

Supervised deep learning-based methods yield accurate results for medical image segmentation. However, they require large labeled datasets for this, and obtaining them is a laborious task that requires clinical expertise. Semi/selfsupervised learning-based approaches address this limitation by exploiting unlabeled data along with limited annotated data. Recent self-supervised learning methods use contrastive loss to learn good global level representations from unlabeled images and achieve high performance in classification tasks on popular natural image datasets like ImageNet. In pixel-level prediction tasks such as segmentation, it is crucial to also learn good local level representations along with global representations to achieve better accuracy. However, the impact of the existing local contrastive loss-based methods remains limited for learning good local representations because similar and dissimilar local regions are defined based on random augmentations and spatial proximity; not based on the semantic label of local regions due to lack of large-scale expert annotations in the semi/self-supervised setting. In this paper, we propose a local contrastive loss to learn good pixel level features useful for segmentation by exploiting semantic label information obtained from pseudo-labels of unlabeled images alongside limited annotated images. In particular, we define the proposed loss to encourage similar representations for the pixels that have the same pseudo-label/ label while being dissimilar to the representation of pixels with different pseudo-label/label in the dataset. We perform pseudo-label based self-training and train the network by jointly optimizing the proposed contrastive loss on both labeled and unlabeled sets and segmentation loss on only the limited labeled set. We evaluated the proposed approach on three public medical datasets of cardiac and prostate anatomies, and obtain high segmentation performance with a limited labeled set of one or two 3D volumes. Extensive comparisons with the state-of-the-art semi-supervised and data augmentation methods and concurrent contrastive learning methods demonstrate the substantial improvement achieved by the proposed method. The code is made publicly available at https://github.com/krishnabits001/pseudo_label_contrastive_training.

show abstract

“…Segsort [37] predicts segmentation by learning to group super-pixels of similar appearance and context from static images. Later work [38] contrasts holistic mask proposals obtained from traditional bottom-up grouping.…”

Section: Related Workmentioning

confidence: 99%

The Emergence of Objectness: Learning Zero-Shot Segmentation from Videos

Liu¹,

Wu²,

Yu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Humans can easily segment moving objects without knowing what they are. That objectness could emerge from continuous visual observations motivates us to model grouping and movement concurrently from unlabeled videos. Our premise is that a video has different views of the same scene related by moving components, and the right region segmentation and region flow would allow mutual view synthesis which can be checked from the data itself without any external supervision. Our model starts with two separate pathways: an appearance pathway that outputs feature-based region segmentation for a single image, and a motion pathway that outputs motion features for a pair of images. It then binds them in a conjoint representation called segment flow that pools flow offsets over each region and provides a gross characterization of moving regions for the entire scene. By training the model to minimize view synthesis errors based on segment flow, our appearance and motion pathways learn region segmentation and flow estimation automatically without building them up from low-level edges or optical flows respectively. Our model demonstrates the surprising emergence of objectness in the appearance pathway, surpassing prior works on zero-shot object segmentation from an image, moving object segmentation from a video with unsupervised test-time adaptation, and semantic image segmentation by supervised fine-tuning. Our work is the first truly end-to-end zero-shot object segmentation from videos. It not only develops generic objectness for segmentation and tracking, but also outperforms prevalent image-based contrastive learning methods without augmentation engineering. * Equal contribution. Work done when Runtao was a StarBridge intern at MSRA. Preprint. Under review.

show abstract

Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals

Cited by 18 publications

References 68 publications

Analyzing Multispectral Satellite Imagery of South American Wildfires Using Deep Learning

Analyzing Multispectral Satellite Imagery of South American Wildfires Using Deep Learning

Local contrastive loss with pseudo-label based self-training for semi-supervised medical image segmentation

The Emergence of Objectness: Learning Zero-Shot Segmentation from Videos

Contact Info

Product

Resources

About