PatchNR: Learning from Small Data by Patch Normalizing Flow Regularization

Altekrüger, Fabian; Denker, Alexander; Hagemann, Paul; Hertrich, Johannes; Maaß, Peter; Steidl, Gabriele

doi:10.48550/arxiv.2205.12021

Cited by 5 publications

(13 citation statements)

References 50 publications

(93 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…If the "best" is understood as "on average" based on some training data, i.e. bilevel scheme (5) with l PSNR and M > 1, we denote these parameters as λ P or Λ P. In that case the test image is not part of the training data.…”

Section: Notations On the Different Regularization Weights And Corres...mentioning

confidence: 99%

“…see e.g. [5,54] for more details. Consequently, we cannot use Algorithm 2 for reconstruction and a reformulation of Algorithm 1 for this data-discrepancy does not lead to a closed form.…”

Section: Problem Formulationmentioning

confidence: 99%

“…Here, (z i , x i true ) M i=1 are M pairs of measured data and corresponding ground truth, and l is a suitable upper level objective. For instance, in the case where l(x 1 , x 2 ) = l PSNR (x 1 , x 2 ) := x 1 − x 2 2 2 , the bilevel problem (5) aims to compute the parameters Λ which are "on the average the best ones" (i.e. PSNR-maximizing), for the given M data pairs.…”

Section: Introductionmentioning

confidence: 99%

“…More precisely, given again M pairs of measured data and corresponding ground truth images (z i , x i true ) M i=1 , during the first part of the offline phase, a corresponding family of optimal regularization parameters (λ i ) M i=1 is computed, e.g. by employing a scheme like (5) separately for each i. Then, in the second part of the offline phase, using the training data D = {(λ i , z i ) M i=1 }, the parameters Θ of a NN N Θ are learned by minimizing…”

Section: Introductionmentioning

confidence: 99%

“…is solved by an appropriate algorithm. The idea is that, due to the good generalizability and adaptability of NNs on unseen data, the computed regularization parameter λ Θ = N Θ (z test ) will be better adapted to z test than the "average" λ of the (scalar parameter version of) bilevel optimization approach (5). The authors in [4] apply this pipeline to learn scalar regularization parameters for computerized tomography reconstruction and image deblurring.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Learning Regularization Parameter-Maps for Variational Image Reconstruction using Deep Neural Networks and Algorithm Unrolling

Kofler¹,

Altekrüger²,

Ba³

et al. 2023

Preprint

View full text Add to dashboard Cite

We introduce a method for fast estimation of data-adapted, spatio-temporally dependent regularization parameter-maps for variational image reconstruction, focusing on total variation (TV)-minimization. Our approach is inspired by recent developments in algorithm unrolling using deep neural networks (NNs), and relies on two distinct sub-networks. The first sub-network estimates the regularization parameter-map from the input data. The second sub-network unrolls T iterations of an iterative algorithm which approximately solves the corresponding TVminimization problem incorporating the previously estimated regularization parameter-map. The overall network is trained end-to-end in a supervised learning fashion using pairs of cleancorrupted data but crucially without the need of having access to labels for the optimal regularization parameter-maps. We prove consistency of the unrolled scheme by showing that the unrolled energy functional used for the supervised learning Γ-converges as T tends to infinity, to the corresponding functional that incorporates the exact solution map of the TV-minimization problem. We apply and evaluate our method on a variety of large scale and dynamic imaging problems in which the automatic computation of such parameters has been so far challenging: 2D dynamic cardiac MRI reconstruction, quantitative brain MRI reconstruction, low-dose CT and dynamic image denoising. The proposed method consistently improves the TV-reconstructions using scalar parameters and the obtained parameter-maps adapt well to each imaging problem and data by leading to the preservation of detailed features. Although the choice of the regularization parameter-maps is data-driven and based on NNs, the proposed algorithm is entirely interpretable since it inherits the properties of the respective iterative reconstruction method from which the network is implicitly defined.

show abstract

Section: Notations On the Different Regularization Weights And Corres...mentioning

confidence: 99%

“…see e.g. [5,54] for more details. Consequently, we cannot use Algorithm 2 for reconstruction and a reformulation of Algorithm 1 for this data-discrepancy does not lead to a closed form.…”

Section: Problem Formulationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Learning Regularization Parameter-Maps for Variational Image Reconstruction using Deep Neural Networks and Algorithm Unrolling

Kofler¹,

Altekrüger²,

Ba³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

Conditional Injective Flows for Bayesian Imaging

Khorashadizadeh

Kothari

Salsi

et al. 2023

IEEE Trans. Comput. Imaging

View full text Add to dashboard Cite

Deep learning is the current de facto state of the art in tomographic imaging. A common approach is to feed the result of a simple inversion, for example the backprojection, to a convolutional neural network (CNN) which then computes the reconstruction. Despite strong results on "in-distribution" test data similar to the training data, backprojection from sparse-view data delocalizes singularities, so these approaches require a large receptive field to perform well. As a consequence, they overfit to certain global structures which leads to poor generalization on out-of-distribution (OOD) samples. Moreover, their memory complexity and training time scale unfavorably with image resolution, making them impractical for application at realistic clinical resolutions, especially in 3D: a standard U-Net requires a substantial 140GB of memory and 2600 seconds per epoch on a research-grade GPU when training on 1024 × 1024 images. In this paper, we introduce GLIMPSE, a local processing neural network for computed tomography which reconstructs a pixel value by feeding only the measurements associated with the neighborhood of the pixel to a simple MLP. While achieving comparable or better performance with successful CNNs like the U-Net on in-distribution test data, GLIMPSE significantly outperforms them on OOD samples while maintaining a memory footprint almost independent of image resolution; 5GB memory suffices to train on 1024 × 1024 images. Further, we built GLIMPSE to be fully differentiable, which enables feats such as recovery of accurate projection angles if they are out of calibration.

show abstract

Learning Regularization Parameter-Maps for Variational Image Reconstruction Using Deep Neural Networks and Algorithm Unrolling

Kofler,

Altekrüger,

Antarou Ba

et al. 2023

SIAM J. Imaging Sci.

View full text Add to dashboard Cite

We introduce a method for the fast estimation of data-adapted, spatially and temporally dependent regularization parameter-maps for variational image reconstruction, focusing on total variation (TV)-minimization. The proposed approach is inspired by recent developments in algorithm unrolling using deep neural networks (NNs), and relies on two distinct sub-networks. The first sub-network estimates the regularization parameter-map from the input data. The second sub-network unrolls T iterations of an iterative algorithm which approximately solves the corresponding TV-minimization problem incorporating the previously estimated regularization parameter-map. The overall network is then trained end-to-end in a supervised learning fashion using pairs of clean-corrupted data but crucially without the need of having access to labels for the optimal regularization parameter-maps. We first prove consistency of the unrolled scheme by showing that the unrolled minimizing energy functional used for the supervised learning Γ-converges as T tends to infinity, to the corresponding functional that incorporates the exact solution map of the TV-minimization problem. Then, we apply and evaluate the proposed method on a variety of large scale and dynamic imaging problems with retrospectively simulated measurement data for which the automatic computation of such regularization parameters has been so far challenging using the state-of-the-art methods: a 2D dynamic cardiac MRI reconstruction problem, a quantitative brain MRI reconstruction problem, a low-dose

show abstract

PatchNR: Learning from Small Data by Patch Normalizing Flow Regularization

Cited by 5 publications

References 50 publications

Learning Regularization Parameter-Maps for Variational Image Reconstruction using Deep Neural Networks and Algorithm Unrolling

Learning Regularization Parameter-Maps for Variational Image Reconstruction using Deep Neural Networks and Algorithm Unrolling

Conditional Injective Flows for Bayesian Imaging

Learning Regularization Parameter-Maps for Variational Image Reconstruction Using Deep Neural Networks and Algorithm Unrolling

Contact Info

Product

Resources

About