Enhancing Diffusion-Based Image Synthesis with Robust Classifier Guidance

Abstract

Denoising diffusion probabilistic models (DDPMs) are a recent family of generative models that achieve state-of-the-art results. In order to obtain class-conditional generation, it was suggested to guide the diffusion process by gradients from a time-dependent classifier. While the idea is theoretically sound, deep learning-based classifiers are infamously susceptible to gradient-based adversarial attacks. Therefore, while traditional classifiers may achieve good accuracy scores, their gradients are possibly unreliable and might hinder the improvement of the generation results. Recent work discovered that adversarially robust classifiers exhibit gradients that are aligned with human perception, and these could better guide a generative process towards semantically meaningful images. We utilize this observation by defining and training a timedependent adversarially robust classifier and use it as guidance for a generative diffusion model. In experiments on the highly challenging and diverse ImageNet dataset, our scheme introduces significantly more intelligible intermediate gradients, better alignment with theoretical findings, as well as improved generation results under several evaluation metrics. Furthermore, we conduct an opinion survey whose findings indicate that human raters prefer our method's results.

Image Denoising: The Deep Learning Revolution and Beyond -- A Survey Paper --

“…The ALD algorithm sparked a wave of related works [261,120,262,208,287,68,122,143,121] that continually improved the performance of these generative diffusion models, eventually surpassing that of GANs [68]. We show some of their results in Figure 8.2.…”

“…This started with the surprising idea that a good performing denoiser can serve as a prior, offering a highly effective regularization to inverse problems [295,231,28,139,283,268,192,280,49,55]. This continued with the discovery that such denoisers can also be used for randomly synthesizing images by offering a practical sampling from the prior distribution of images, this way posing a potent competition to Generative Adversarial Networks (GANs) and other image generation methods [260,261,262,120,287,68,122,143,121].…”

Imagic: Text-Based Real Image Editing with Diffusion Models

“…Diffusion models [18,51,53,58] are a family of generative models that has recently gained traction, as they advanced the state-of-the-art in image generation [12,26,54,57], and have been deployed in various downstream applications such as image restoration [25,45], adversarial purification [10,34], image compression [55], image classification [61], and others [14,27,37,48,59].…”