Label-Efficient Semantic Segmentation with Diffusion Models

Abstract

Denoising diffusion probabilistic models have recently received much research attention since they outperform alternative approaches, such as GANs, and currently provide state-of-the-art generative performance. The superior performance of diffusion models has made them an appealing tool in several applications, including inpainting, super-resolution, and semantic editing. In this paper, we demonstrate that diffusion models can also serve as an instrument for semantic segmentation, especially in the setup when labeled data is scarce. In particular, for several pretrained diffusion models, we investigate the intermediate activations from the networks that perform the Markov step of the reverse diffusion process. We show that these activations effectively capture the semantic information from an input image and appear to be excellent pixel-level representations for the segmentation problem. Based on these observations, we describe a simple segmentation method, which can work even if only a few training images are provided. Our approach significantly outperforms the existing alternatives on several datasets for the same amount of human supervision. The source code of the project is publicly available.

Improving Sample Quality of Diffusion Models Using Self-Attention Guidance

“…Besides, motivated by the success of the diffusion models in the generation tasks, there are some efforts to utilize the diffusion models in other tasks, such as semantic segmentation. Brempong et al (Brempong et al 2022) showed that the denoising pretraining boosts the performance of the semantic segmentation task, and Baranchuk et al (Baranchuk et al 2021) proposed a label-efficient strategy for semantic segmentation using the UNet (Ronneberger, Fischer, and Brox 2015) representation of DDPM (Ho, Jain, and Abbeel 2020).…”

“…Although some literature utilized UNet (Ronneberger, Fischer, and Brox 2015) feature maps of diffusion models for label-efficient semantic segmentation (Baranchuk et al 2021;Brempong et al 2022), analyzing and utilizing the self-attention maps inside diffusion models remain underexplored. In this section, by conducting experiments with those self-attention maps, we carefully examine the regions to which the self-attention in a diffusion model attends.…”

“…Finally, the comparison of the extraction layers of the attention map is shown in Table 6. Following (Baranchuk et al 2021) that used the decoder representations for semantic segmentation, we select the last self-attention layers of each resolution from the decoder, i.e., 8×8, 16×16, and 32×32. Regardless of the extracted layer, the performance is consistently improved, while the result of the 32 × 32 resolution gives the best FID and IS.…”

“…maps of DINO (Caron et al 2021) to extract object masks in unsupervised semantic segmentation settings. However, while there are some attempts to use the UNet (Ronneberger, Fischer, and Brox 2015) representation of a diffusion model (Baranchuk et al 2021;Brempong et al 2022), the internal self-attention maps of diffusion models are under-explored. Therefore, in this paper, we first investigate the self-attention maps within the black-boxed diffusion models and build hypotheses about them.…”

Diffusion Models for Implicit Image Segmentation Ensembles

“…While some new work applies diffusion models on tasks such as image-to-image translation (Sasaki et al, 2021), style transfer (Choi et al, 2021), or inpainting tasks (Saharia et al, 2021), so far there is only very little work about semantic segmentation. Recently, one approach to perform semantic segmentation with a diffusion model was proposed by (Baranchuk et al, 2021). A DDPM is trained to reconstruct the image that should be segmented.…”