SegDiff: Image Segmentation with Diffusion Probabilistic Models

Amit, Tomer; Nachmani, Eliya; Shaharbany, Tal; Wolf, Lior

doi:10.48550/arxiv.2112.00390

Cited by 41 publications

(50 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nevertheless, these iterative algorithms are still considerably slower than GANs, so substantial work has been invested in improving their speed without compromising significantly on generation quality [258,135,247], often achieving impressive speedup levels. Diffusion models have since become ubiquitous in many applications [142,209,21,116,6,253,254,144], prompting researchers to prepare surveys of their impact on the image processing field and beyond [315,60,36]. Figure 8.1: Temporal steps along 3 independent synthesis paths of the Annealed Langevin Dynamics [260] algorithm, using a denoiser [261] trained on LSUN bedroom [319] images.…”

Section: Regularization By Denoising (Red)mentioning

confidence: 99%

Image Denoising: The Deep Learning Revolution and Beyond -- A Survey Paper --

Elad¹,

Kawar²,

Vaksman³

2023

Preprint

View full text Add to dashboard Cite

Image denoising -removal of additive white Gaussian noise from an image -is one of the oldest and most studied problems in image processing. An extensive work over several decades has led to thousands of papers on this subject, and to many well-performing algorithms for this task. Indeed, ten years ago, these achievements have led some researchers to suspect that "Denoising is Dead", in the sense that all that can be achieved in this domain has already been obtained. However, this turned out to be far from the truth, with the penetration of deep learning (DL) into the realm of image processing. The era of DL brought a revolution to image denoising, both by taking the lead in today's ability for noise suppression in images, and by broadening the scope of denoising problems being treated. Our paper starts by describing this evolution, highlighting in particular the tension and synergy that exist between classical approaches and modern Artificial Intelligence (AI) alternatives in design of image denoisers.The recent transitions in the field of image denoising go far beyond the ability to design better denoisers. In the second part of this paper we focus on recently discovered abilities and prospects of image denoisers. We expose the possibility of using image denoisers for service of other problems, such as regularizing general inverse problems and serving as the prime engine in diffusion-based image synthesis. We also unveil the (strange?) idea that denoising and other inverse problems might not have a unique solution, as common algorithms would have us believe. Instead, we describe constructive ways to produce randomized and diverse high perceptual quality results for inverse problems, all fueled by the progress that DL brought to image denoising. This is a survey paper, and its prime goal is to provide a broad view of the history of the field of image denoising and closely related topics in image processing. Our aim is to give a better context to recent discoveries, and to the influence of the AI revolution in our domain.

show abstract

Section: Regularization By Denoising (Red)mentioning

confidence: 99%

Image Denoising: The Deep Learning Revolution and Beyond -- A Survey Paper --

Elad¹,

Kawar²,

Vaksman³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…To date, diffusion models have been found to be useful in a wide variety of areas, ranging from generative modeling tasks such as image generation [23], image super-resolution [24], image inpainting [25] to discriminative tasks such as image segmentation [26], classification [27], and anomaly detection [28]. Recently, the medical imaging community has witnessed exponential growth in the number of diffusion-based techniques (see Figure 4).…”

Section: Denoising Diffusion Modelsmentioning

confidence: 99%

Diffusion Models for Medical Image Analysis: A Comprehensive Survey

Kazerouni¹,

Aghdam²,

Heidari³

et al. 2022

Preprint

View full text Add to dashboard Cite

Denoising diffusion models, a class of generative models, have garnered immense interest lately in various deep-learning problems. A diffusion probabilistic model defines a forward diffusion stage where the input data is gradually perturbed over several steps by adding Gaussian noise and then learns to reverse the diffusion process to retrieve the desired noise-free data from noisy data samples. Diffusion models are widely appreciated for their strong mode coverage and quality of the generated samples in spite of their known computational burdens. Capitalizing on the advances in computer vision, the field of medical imaging has also observed a growing interest in diffusion models. With the aim of helping the researcher navigate this profusion, this survey intends to provide a comprehensive overview of diffusion models in the discipline of medical image analysis. Specifically, we start with an introduction to the solid theoretical foundation and fundamental concepts behind diffusion models and the three generic diffusion modeling frameworks, namely, diffusion probabilistic models, noise-conditioned score networks, and stochastic differential equations. Then, we provide a systematic taxonomy of diffusion models in the medical domain and propose a multi-perspective categorization based on their application, imaging modality, organ of interest, and algorithms. To this end, we cover extensive applications of diffusion models in the medical domain, including segmentation, anomaly detection, image-to-image translation, 2/3D generation, reconstruction, denoising, and other medically-related challenges. Furthermore, we emphasize the practical use case of some selected approaches, and then we discuss the limitations of the diffusion models in the medical domain and propose several directions to fulfill the demands of this field. Finally, we gather the overviewed studies with their available open-source implementations at our GitHub 1 . We aim to update the relevant latest papers within it regularly.

show abstract

“…Existing generalist models Image Prediction Groundtruth are based on autoregressive models, while our work is based on Bit Diffusion [12,23,50,51]. Diffusion models have been applied to semantic segmentation, directly [1,27,63] or in-directly [2,4]. However none of these methods model segmentation masks as discrete/categorical tokens, nor are their models capable of video segmentation.…”

Section: Related Workmentioning

confidence: 99%

A Generalist Framework for Panoptic Segmentation of Images and Videos

Chen¹,

Li²,

Saxena³

et al. 2022

Preprint

View full text Add to dashboard Cite

Panoptic segmentation assigns semantic and instance ID labels to every pixel of an image. As permutations of instance IDs are also valid solutions, the task requires learning of high-dimensional one-to-many mapping. As a result, state-of-the-art approaches use customized architectures and task-specific loss functions. We formulate panoptic segmentation as a discrete data generation problem, without relying on inductive bias of the task. A diffusion model based on analog bits [12] is used to model panoptic masks, with a simple, generic architecture and loss function. By simply adding past predictions as a conditioning signal, our method is capable of modeling video (in a streaming setting) and thereby learns to track object instances automatically. With extensive experiments, we demonstrate that our generalist approach can perform competitively to stateof-the-art specialist methods in similar settings.

show abstract

SegDiff: Image Segmentation with Diffusion Probabilistic Models

Cited by 41 publications

References 42 publications

Image Denoising: The Deep Learning Revolution and Beyond -- A Survey Paper --

Image Denoising: The Deep Learning Revolution and Beyond -- A Survey Paper --

Diffusion Models for Medical Image Analysis: A Comprehensive Survey

A Generalist Framework for Panoptic Segmentation of Images and Videos

Contact Info

Product

Resources

About