Conditional Image Generation with Score-Based Diffusion Models

Batzolis, Georgios; Stanczuk, Jan; Schönlieb, Carola‐Bibiane; Etmann, Christian

doi:10.48550/arxiv.2111.13606

Cited by 19 publications

(29 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Denoising diffusion models [60,64] have seen great success on a wide variety of different challenges, ranging from image2image translation tasks like inpainting, colorisation, image upscaling, uncropping [6,26,41,42,50,53,57,59], audio generation [11,28,33,35,38,48,67,80], text-based image generation [4,21,23,46,51,55,58], video generation [24,27,82,86], and many others. For a thorough review on diffusion models and all of their recent applications, we recommend [81].…”

Section: Diffusion Modelsmentioning

confidence: 99%

Speech Driven Video Editing via an Audio-Conditioned Diffusion Model

Bigioi¹,

Basak²,

Jordan³

et al. 2023

Preprint

View full text Add to dashboard Cite

In this paper we propose a method for end-to-end speech driven video editing using a denoising diffusion model. Given a video of a person speaking, we aim to re-synchronise the lip and jaw motion of the person in response to a separate auditory speech recording without relying on intermediate structural representations such as facial landmarks or a 3D face model. We show this is possible by conditioning a denoising diffusion model with audio spectral features to generate synchronised facial motion. We achieve convincing results on the task of unstructured single-speaker video editing, achieving a word error rate of 45% using an off the shelf lip reading model. We further demonstrate how our approach can be extended to the multi-speaker domain. To our knowledge, this is the first work to explore the feasibility of applying denoising diffusion models to the task of audio-driven video editing. 1

show abstract

Section: Diffusion Modelsmentioning

confidence: 99%

Speech Driven Video Editing via an Audio-Conditioned Diffusion Model

Bigioi¹,

Basak²,

Jordan³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Diffusion models surpass the original SOTA: Generative Adversarial Network (GAN) in image generation tasks and excel in many application areas such as computer vision, NLP, waveform signal processing, multimodal modeling, molecular graph modeling, time series modeling, adversarial purification, etc. Generative models are used in computer vision to handle various image recovery tasks including super-resolution, restoration, and panning [ 23 , 24 , 25 ], and in NLP to generate character-level text by discrete denoising diffusion probability models (D3PM) [ 26 ]. In addition, in terms of time series data generation, refs.…”

Section: Related Workmentioning

confidence: 99%

A Diffusion Model Based on Network Intrusion Detection Method for Industrial Cyber-Physical Systems

Tang

et al. 2023

Sensors

View full text Add to dashboard Cite

Industrial Cyber-Physical Systems (ICPS) connect intelligent manufacturing equipment equipped with sensors, wireless and RFID communication technologies through data interaction, which makes the interior of the factory, even between factories, become a whole. However, intelligent factories will suffer information leakage and equipment damage when being attacked by ICPS intrusion. Therefore, the network security of ICPS cannot be ignored, and researchers have conducted in-depth research on network intrusion detection for ICPS. Though machine learning and deep learning methods are often used for network intrusion detection, the problem of data imbalance can cause the model to pay attention to the misclassification cost of the prevalent class, but ignore that of the rare class, which seriously affects the classification performance of network intrusion detection models. Considering the powerful generative power of the diffusion model, we propose an ICPS Intrusion Detection system based on the Diffusion model (IDD). Firstly, data corresponding to the rare class is generated by the diffusion model, which makes the training dataset of different classes balanced. Then, the improved BiLSTM classification network is trained on the balanced training set. Extensive experiments are conducted to show that the IDD method outperforms the existing baseline method on several available datasets.

show abstract

“…They all follow similar processes: a forward process which gradually adds noise to clean samples drawn from a prior distribution and a reverse process which reverses the corruption process to recover plausible samples from noise. Diffusion models have been successful at many challenging image-based tasks such as unconditional image generation [64], inpainting [66,53,6,38], colorization [66,38], image segmentation [5,2], and medical imaging [72,84]. We refer the reader to a survey [14] for more applications.…”

Section: Related Workmentioning

confidence: 99%

Diffusion in the Dark: A Diffusion Model for Low-Light Text Recognition

Nguyen¹,

Chan²,

Bergman³

et al. 2023

Preprint

View full text Add to dashboard Cite

Images are indispensable for the automation of highlevel tasks, such as text recognition. Low-light conditions pose a challenge for these high-level perception stacks, which are often optimized on well-lit, artifact-free images. Reconstruction methods for low-light images can produce well-lit counterparts, but typically at the cost of highfrequency details critical for downstream tasks. We propose Diffusion in the Dark (DiD), a diffusion model for lowlight image reconstruction that provides qualitatively competitive reconstructions with that of SOTA, while preserving high-frequency details even in extremely noisy, dark conditions. We demonstrate that DiD, without any task-specific optimization, can outperform SOTA low-light methods in low-light text recognition on real images, bolstering the potential of diffusion models for ill-posed inverse problems.

show abstract

Conditional Image Generation with Score-Based Diffusion Models

Cited by 19 publications

References 0 publications

Speech Driven Video Editing via an Audio-Conditioned Diffusion Model

Speech Driven Video Editing via an Audio-Conditioned Diffusion Model

A Diffusion Model Based on Network Intrusion Detection Method for Industrial Cyber-Physical Systems

Diffusion in the Dark: A Diffusion Model for Low-Light Text Recognition

Contact Info

Product

Resources

About