Generic Image Restoration with Flow Based Priors

Helminger, Leonhard; Bernasconi, M.; Djelouah, Abdelaziz; Groß, Markus; Schroers, Christopher

doi:10.1109/cvprw53098.2021.00043

Cited by 11 publications

(13 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this work we explored and clarified the tight relationships between joint map-x-z estimation, splitting and continuation schemes and the more common map-z estimator in the context of inverse problems with a generative prior. On the other hand map-x estimators (which are otherwise standard in bayesian imaging) remained largely unexplored in the context of generative priors, due to the optimization challenges they impose, until the recent work of Helminger et al [24], Whang et al [57] showed that a normalizing flow-based generative model allows to overcome those challenges and deems this problem tractable. Similarly Oberlin and Verm [35] use Glow (an invertible normalizing flow) to compare synthesis-based and analysis-based reconstructions.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Solving Inverse Problems by Joint Posterior Maximization with Autoencoding Prior

González,

Almansa,

Tan

2021

Preprint

View full text Add to dashboard Cite

In this work we address the problem of solving ill-posed inverse problems in imaging where the prior is a variational autoencoder (VAE). Specifically we consider the decoupled case where the prior is trained once and can be reused for many different log-concave degradation models without retraining. Whereas previous MAP-based approaches to this problem lead to highly non-convex optimization algorithms, our approach computes the joint (space-latent) MAP that naturally leads to alternate optimization algorithms and to the use of a stochastic encoder to accelerate computations. The resulting technique (JPMAP) performs Joint Posterior Maximization using an Autoencoding Prior. We show theoretical and experimental evidence that the proposed objective function is quite close to bi-convex. Indeed it satisfies a weak bi-convexity property which is sufficient to guarantee that our optimization scheme converges to a stationary point. We also highlight the importance of correctly training the VAE using a denoising criterion, in order to ensure that the encoder generalizes well to out-of-distribution images, without affecting the quality of the generative model. This simple modification is key to providing robustness to the whole procedure. Finally we show how our joint MAP methodology relates to more common MAP approaches, and we propose a continuation scheme that makes use of our JPMAP algorithm to provide more robust MAP estimates. Experimental results also show the higher quality of the solutions obtained by our JPMAP approach with respect to other non-convex MAP approaches which more often get stuck in spurious local optima.

show abstract

Section: Discussionmentioning

confidence: 99%

“…• the inversion of G, and • the hard constraint x ∈ M. These operations are are all memory and/or computationally intensive, except when they are partially addressed by the use of a normalizing flow like in [24,57].…”

mentioning

confidence: 99%

Solving Inverse Problems by Joint Posterior Maximization with Autoencoding Prior

González,

Almansa,

Tan

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Learned importance sampling has also been studied for complex luminaires rendering [ZBX * 21]. Besides these tasks, NFs have been utilized in computer graphics and vision for image [KD18, WZY22] and video [KBE * 19] generation, compression [HDGS20], super‐resolution [LDVGT20], domain translation [GCS * 20], and uncertainty quantification [WLM * 22, SFMP20]. We build upon Neural Importance Sampling [MMR * 19] and propose a lightweight model for sampling and PDF evaluation of environment maps, as we describe in Section 4.…”

Section: Related Workmentioning

confidence: 99%

NEnv: Neural Environment Maps for Global Illumination

Rodríguez-Pardo

Fabre

Garcés

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

Environment maps are commonly used to represent and compute far‐field illumination in virtual scenes. However, they are expensive to evaluate and sample from, limiting their applicability to real‐time rendering. Previous methods have focused on compression through spherical‐domain approximations, or on learning priors for natural, day‐light illumination. These hinder both accuracy and generality, and do not provide the probability information required for importance‐sampling Monte Carlo integration. We propose NEnv, a deep‐learning fully‐differentiable method, capable of compressing and learning to sample from a single environment map. NEnv is composed of two different neural networks: A normalizing flow, able to map samples from uniform distributions to the probability density of the illumination, also providing their corresponding probabilities; and an implicit neural representation which compresses the environment map into an efficient differentiable function. The computation time of environment samples with NEnv is two orders of magnitude less than with traditional methods. NEnv makes no assumptions regarding the content (i.e. natural illumination), thus achieving higher generality than previous learning‐based approaches. We share our implementation and a diverse dataset of trained neural environment maps, which can be easily integrated into existing rendering engines.

show abstract

“…On the other hand, there is another notable class of approaches recently put forward is based on flow-based invertible neural networks. These generative networks have invertible architectures and a latent space with the same size of the image space, and thus have zero representation error [1,3,22,20,28,34,33,13]. Both these type of approaches attempt to limit the representation error during training or inversion.…”

Section: Related Workmentioning

confidence: 99%

Regularized Training of Intermediate Layers for Generative Models for Inverse Problems

Gunn¹,

Cocola²,

Hand³

2022

Preprint

View full text Add to dashboard Cite

Generative Adversarial Networks (GANs) have been shown to be powerful and flexible priors when solving inverse problems. One challenge of using them is overcoming representation error, the fundamental limitation of the network in representing any particular signal. Recently, multiple proposed inversion algorithms reduce representation error by optimizing over intermediate layer representations. These methods are typically applied to generative models that were trained agnostic of the downstream inversion algorithm. In our work, we introduce a principle that if a generative model is intended for inversion using an algorithm based on optimization of intermediate layers, it should be trained in a way that regularizes those intermediate layers. We instantiate this principle for two notable recent inversion algorithms: Intermediate Layer Optimization and the Multi-Code GAN prior. For both of these inversion algorithms, we introduce a new regularized GAN training algorithm and demonstrate that the learned generative model results in lower reconstruction errors across a wide range of under sampling ratios when solving compressed sensing, inpainting, and super-resolution problems.

show abstract

Generic Image Restoration with Flow Based Priors

Cited by 11 publications

References 14 publications

Solving Inverse Problems by Joint Posterior Maximization with Autoencoding Prior

Solving Inverse Problems by Joint Posterior Maximization with Autoencoding Prior

NEnv: Neural Environment Maps for Global Illumination

Regularized Training of Intermediate Layers for Generative Models for Inverse Problems

Contact Info

Product

Resources

About