Generative Modeling with Denoising Auto-Encoders and Langevin Sampling

Block, Adam; Mroueh, Youssef; Rakhlin, Alexander

doi:10.48550/arxiv.2002.00107

Cited by 11 publications

(21 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, another notable paper close in spirit to our goal is Block et al [2020], which provides a detailed theoretical analysis of a score-matching generative model using Denoising Autoencoders followed by Langevin diffusion. While their work makes generally weaker assumptions and also includes a non-asymptotic analysis of the sampling algorithm, the resulting rates are unsuprisingly cursed by dimension.…”

Section: Related Workmentioning

confidence: 93%

“…Such sampling procedures may be difficult in general, particularly for complex energy landscapes, thus we also consider different estimators based on un-normalized measures which avoid the need of sampling. We focus here on approaches based on minimizing Stein discrepancies Mackey, 2015, Liu andWang, 2016], which have recently been found to be useful in deep generative models [Grathwohl et al, 2020], though we note that alternative approaches may be used, such as score matching [Hyvärinen, 2005, Song and Kingma, 2021, Song and Ermon, 2019, Block et al, 2020.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

On Energy-Based Models with Overparametrized Shallow Neural Networks

Domingo-Enrich,

Bietti,

Vanden-Eijnden

et al. 2021

Preprint

View full text Add to dashboard Cite

Energy-based models (EBMs) are a simple yet powerful framework for generative modeling. They are based on a trainable energy function which defines an associated Gibbs measure, and they can be trained and sampled from via well-established statistical tools, such as MCMC. Neural networks may be used as energy function approximators, providing both a rich class of expressive models as well as a flexible device to incorporate data structure. In this work we focus on shallow neural networks. Building from the incipient theory of overparametrized neural networks, we show that models trained in the so-called "active" regime provide a statistical advantage over their associated "lazy" or kernel regime, leading to improved adaptivity to hidden low-dimensional structure in the data distribution, as already observed in supervised learning. Our study covers both maximum likelihood and Stein Discrepancy estimators, and we validate our theoretical results with numerical experiments on synthetic data.

show abstract

Section: Related Workmentioning

confidence: 93%

Section: Introductionmentioning

confidence: 99%

On Energy-Based Models with Overparametrized Shallow Neural Networks

Domingo-Enrich,

Bietti,

Vanden-Eijnden

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…However, the main difference lies that the drift terms of the Langevin SDEs in (Song & Ermon, 2019;Block et al, 2020) are time-invariant in contrast to the time-varying drift term in our formulation. As shown in Theorem 9, the benefit of the time-varying drift term is essential: the SDE of Schrödinger Bridge runs on a unit time interval [0, 1] will recover the target distribution at the terminal time.…”

Section: Related Workmentioning

confidence: 98%

“…The consistency of the proposed Schrödinger Bridge is mainly based on mild assumptions (such as smoothness and boundedness) without some restricted technical requirements that the target distribution has to be log-concave or fulfill the log-Sobolev inequality (Gao et al, 2020;Arbel et al, 2019;Liutkus et al, 2019;Block et al, 2020).…”

Section: Now We Establish the Consistency Of Our Schrödinger Bridgementioning

confidence: 99%

“…Recently, new generative models have been developed from different perspectives, such as gradient flow in a measure space in which GAN can be covered as a special case (Gao et al, 2019;Arbel et al, 2019) and stochastic differential equations (SDE) (Song & Ermon, 2019;Song et al, 2021b). To push a simple initial distribution to the target one, however, these methods (Gao et al, 2019;Arbel et al, 2019;Liutkus et al, 2019;Song & Ermon, 2019;Block et al, 2020) require the evolving time to go to infinity at the population level. Therefore, these methods require a strong assumption to achieve model consistency: the target must be log-concave or satisfy the log-Sobolev inequality.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Deep Generative Learning via Schrödinger Bridge

Wang,

Jiao,

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose to learn a generative model via entropy interpolation with a Schrödinger Bridge. The generative learning task can be formulated as interpolating between a reference distribution and a target distribution based on the Kullback-Leibler divergence. At the population level, this entropy interpolation is characterized via an SDE on [0, 1] with a time-varying drift term. At the sample level, we derive our Schrödinger Bridge algorithm by plugging the drift term estimated by a deep score estimator and a deep density ratio estimator into the Euler-Maruyama method. Under some mild smoothness assumptions of the target distribution, we prove the consistency of both the score estimator and the density ratio estimator, and then establish the consistency of the proposed Schrödinger Bridge approach. Our theoretical results guarantee that the distribution learned by our approach converges to the target distribution. Experimental results on multimodal synthetic data and benchmark data support our theoretical findings and indicate that the generative model via Schrödinger Bridge is comparable with state-ofthe-art GANs, suggesting a new formulation of generative learning. We demonstrate its usefulness in image interpolation and image inpainting.

show abstract

Homotopic Gradients of Generative Density Priors for MR Image Reconstruction

Quan

Zhou

Zhu

et al. 2021

IEEE Trans. Med. Imaging

View full text Add to dashboard Cite

Deep learning, particularly the generative model, has demonstrated tremendous potential to significantly speed up image reconstruction with reduced measurements recently. Rather than the existing generative models that often optimize the density priors, in this work, by taking advantage of the denoising score matching, homotopic gradients of generative density priors (HGGDP) are proposed for magnetic resonance imaging (MRI) reconstruction. More precisely, to tackle the low-dimensional manifold and low data density region issues in generative density prior, we estimate the target gradients in higher-dimensional space. We train a more powerful noise conditional score network by forming high-dimensional tensor as the network input at the training phase. More artificial noise is also injected in the embedding space. At the reconstruction stage, a homotopy method is employed to pursue the density prior, such as to boost the reconstruction performance. Experiment results imply the remarkable performance of HGGDP in terms of high reconstruction accuracy; only 10% of the k-space data can still generate images of high quality as effectively as standard MRI reconstruction with the fully sampled data.

show abstract

Generative Modeling with Denoising Auto-Encoders and Langevin Sampling

Cited by 11 publications

References 11 publications

On Energy-Based Models with Overparametrized Shallow Neural Networks

On Energy-Based Models with Overparametrized Shallow Neural Networks

Deep Generative Learning via Schrödinger Bridge

Homotopic Gradients of Generative Density Priors for MR Image Reconstruction

Contact Info

Product

Resources

About