Prior Image-Constrained Reconstruction using Style-Based Generative Models

Kelkar, Varun A.; Anastasio, Mark A.

doi:10.48550/arxiv.2102.12525

Cited by 5 publications

(9 citation statements)

References 24 publications

(50 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent methods have been developed for regularizing image reconstruction problems based on GANs such as Compressed Sensing using Generative Models (CSGM) [60] and image-adaptive GAN-based reconstruction methods (IAGAN) [61], [62]. Sty2AmGANs can also be used for prior image-constrained reconstruction [59]. The established SOMs can also be used to produce clean reference images for training deep neural networks for solving other image-processing problems such as image denoising [63], [64] and image super-resolution [65].…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Learning stochastic object models from medical imaging measurements by use of advanced ambient generative adversarial networks

Zhou,

Bhadra,

Brooks

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In order to objectively assess new medical imaging technologies via computer-simulations, it is important to account for all sources of variability that contribute to image data. One important source of variability that can significantly limit observer performance is associated with the variability in the ensemble of objects to-be-imaged. This source of variability can be described by stochastic object models (SOMs), which are generative models that can be employed to sample from a distribution of to-be-virtually-imaged objects. It is generally desirable to establish SOMs from experimental imaging measurements acquired by use of a well-characterized imaging system, but this task has remained challenging. Deep generative neural networks, such as generative adversarial networks (GANs) hold potential for such tasks. To establish SOMs from imaging measurements, an AmbientGAN has been proposed that augments a GAN with a measurement operator. However, the original AmbientGAN could not immediately benefit from modern training procedures and GAN architectures, which limited its ability to be applied to realistically sized medical image data. To circumvent this, in this work, a modified AmbientGAN training strategy is proposed that is suitable for modern progressive or multi-resolution training approaches such as employed in the Progressive Growing of GANs and Style-based GANs. AmbientGANs established by use of the proposed training procedure are systematically validated in a controlled way by use of computer-simulated measurement data corresponding to a stylized imaging system. Finally, emulated single-coil experimental magnetic resonance imaging data are employed to demonstrate the methods under less stylized conditions.

show abstract

Section: Discussionmentioning

confidence: 99%

“…Because the groundtruth objects corresponding to the synthesized images were not accessible in this experimental study, only a subjective visual assessment was performed. The style-based generator used in Sty2AmGAN can provide additional ability to control scale-specific image features [58], [59].…”

Section: Experimental Emulated Single-coil Mri Datamentioning

confidence: 99%

Learning stochastic object models from medical imaging measurements by use of advanced ambient generative adversarial networks

Zhou,

Bhadra,

Brooks

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Aspects of this work were published as a part of the International Conference on Machine Learning (ICML) 2021. 21…”

Section: Disclosurementioning

confidence: 99%

Prior image-based medical image reconstruction using a style-based generative adversarial network

Kelkar¹,

Anastasio²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Computed medical imaging systems require a computational reconstruction procedure for image formation. In order to recover a useful estimate of the object to-be-imaged when the recorded measurements are incomplete, prior knowledge about the nature of object must be utilized. In order to improve the conditioning of an ill-posed imaging inverse problem, deep learning approaches are being actively investigated for better representing object priors and constraints. This work proposes to use a style-based generative adversarial network (StyleGAN) to constrain an image reconstruction problem in the case where additional information in the form of a prior image of the sought-after object is available. An optimization problem is formulated in the intermediate latent-space of a StyleGAN, that is disentangled with respect to meaningful image attributes or "styles", such as the contrast used in magnetic resonance imaging (MRI). Discrepancy between the sought-after and prior images is measured in the disentangled latent-space, and is used to regularize the inverse problem in the form of constraints on specific styles of the disentangled latent-space. A stylized numerical study inspired by MR imaging is designed, where the sought-after and the prior image are structurally similar, but belong to different contrast mechanisms. The presented numerical studies demonstrate the superiority of the proposed approach as compared to classical approaches in the form of traditional metrics.

show abstract

“…In addition to a superior performance in image synthesis, the style-specific control that is gained with a StyleGAN generator can be leveraged to perform meaningful semantic transformations of objects in tomographic imaging applications [31]- [33]. To perform such semantic transformations, an embedding for the given object must be obtained first in the latent space of the StyleGAN.…”

Section: A Salient Features Of the Stylegan Latent Spacementioning

confidence: 99%

“…Second, the tolerance level for data consistency to accept a high-resolution image estimate was chosen in an arbitrary fashion irrespective of the measurement noise distribution, and thus it is difficult to determine the degree to which data consistency is being preserved. Third, while the PULSE method as well as previous studies such as [31], [35], [36] employed the prior assumption that the latent vectors {v i } follow a standard multivariate Gaussian distribution, no rigorous quantitative evaluation was performed to justify the accuracy of this ansatz. While the lack of such quantitative validation may still be acceptable for a computer vision task where the objective is to obtain diverse photo-realistic face images from a given low-resolution image, quantitative assessment of the method is critical for a proper assessment of tomographic imaging systems.…”

Section: B Empirical Sampling With Pulsementioning

confidence: 99%

Mining the manifolds of deep generative models for multiple data-consistent solutions of ill-posed tomographic imaging problems

Bhadra¹,

Villa²,

Anastasio³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Tomographic imaging is in general an ill-posed inverse problem. Typically, a single regularized image estimate of the sought-after object is obtained from tomographic measurements. However, there may be multiple objects that are all consistent with the same measurement data. The ability to generate such alternate solutions is important because it may enable new assessments of imaging systems. In principle, this can be achieved by means of posterior sampling methods. In recent years, deep neural networks have been employed for posterior sampling with promising results. However, such methods are not yet for use with large-scale tomographic imaging applications. On the other hand, empirical sampling methods may be computationally feasible for large-scale imaging systems and enable uncertainty quantification for practical applications. Empirical sampling involves solving a regularized inverse problem within a stochastic optimization framework in order to obtain alternate data-consistent solutions. In this work, we propose a new empirical sampling method that computes multiple solutions of a tomographic inverse problem that are consistent with the same acquired measurement data. The method operates by repeatedly solving an optimization problem in the latent space of a style-based generative adversarial network (StyleGAN), and was inspired by the Photo Upsampling via Latent Space Exploration (PULSE) method that was developed for super-resolution tasks. The proposed method is demonstrated and analyzed via numerical studies that involve two stylized tomographic imaging modalities. These studies establish the ability of the method to perform efficient empirical sampling and uncertainty quantification.

show abstract

Prior Image-Constrained Reconstruction using Style-Based Generative Models

Cited by 5 publications

References 24 publications

Learning stochastic object models from medical imaging measurements by use of advanced ambient generative adversarial networks

Learning stochastic object models from medical imaging measurements by use of advanced ambient generative adversarial networks

Prior image-based medical image reconstruction using a style-based generative adversarial network

Mining the manifolds of deep generative models for multiple data-consistent solutions of ill-posed tomographic imaging problems

Contact Info

Product

Resources

About