Neumann Networks for Linear Inverse Problems in Imaging

Abstract

Many challenging image processing tasks can be described by an ill-posed linear inverse problem: deblurring, deconvolution, inpainting, compressed sensing, and superresolution all lie in this framework. Traditional inverse problem solvers minimize a cost function consisting of a data-fit term, which measures how well an image matches the observations, and a regularizer, which reflects prior knowledge and promotes images with desirable properties like smoothness. Recent advances in machine learning and image processing have illustrated that it is often possible to learn a regularizer from training data that can outperform more traditional regularizers. We present an end-to-end, data-driven method of solving inverse problems inspired by the Neumann series, which we call a Neumann network. Rather than unroll an iterative optimization algorithm, we truncate a Neumann series which directly solves the linear inverse problem with a data-driven nonlinear regularizer. The Neumann network architecture outperforms traditional inverse problem solution methods, model-free deep learning approaches, and state-of-the-art unrolled iterative methods on standard datasets. Finally, when the images belong to a union of subspaces and under appropriate assumptions on the forward model, we prove there exists a Neumann network configuration that well-approximates the optimal oracle estimator for the inverse problem and demonstrate empirically that the trained Neumann network has the form predicted by theory. Learning to RegularizeIn this paper we consider solving linear inverse problems in imaging in which a p-pixel image, β ∈ R p (in vectorized form), is observed via m noisy linear projections as y = Xβ + , where y, ∈ R m and X ∈ R m×p . This general model is used throughout computational imaging, from basic image restoration tasks like deblurring, super-resolution, and image inpainting [1], to a wide variety of tomographic imaging applications, including common types of magnetic resonance imaging [2], X-ray computed tomography [3], radar imaging [4], among others [5]. The task of estimating * D. Gilton is with the

Compressive Sensing Spectroscopy Using a Residual Convolutional Neural Network

“…To recover multi-modal data, a reconstruction framework is proposed in [42] that uses side information in unrolled optimization. Unrolled optimization approaches using deep learning were proposed in [43,44]. Deep-learning architectures were used to train hyper-parameters, such as a gradient regularizer and a step size.…”

“…Compared with CNN, ResCNN shows significant improvement in reconstruction performance and converges faster than CNN. In future work, we will explore compression approaches [40] and unrolled optimization approaches [43,44] for generating a sparsifying basis Φ from the training dataset to fully represent spectra without loss of spectral features.…”

Leveraging spatial textures, through machine learning, to identify aerosols and distinct cloud types from multispectral observations

“…exponentially with the size of the input images (Gilton et al, 2020). In this paper the training dataset is relatively small since it is time-consuming to hand-label MODIS-VIIRS images and for optically thick aerosols there are not enough events in the observations.…”

“…For example, neural networks have been employed to (1) approximate computationally demanding radiative transfer models to decrease computation time (Boukabara et al, 2019;Blackwell, 2005;Takenaka et al, 2011), (2) infer tropical cyclone intensity from microwave imagery (Wimmers et al, 2019), (3) infer cloud vertical structures and cirrus or high-altitude cloud optical depths from MODIS imagery (Leinonen et al, 2019;Minnis et al, 2016), and (4) predict the formation of large hailstones from land-based radar imagery (Gagne et al, 2019). Specific to cloud and volcanic ash detection from radiometer images, Bayesian inference has been employed where the posterior distribution functions were empirically generated using hand-labeled (Pavolonis et al, 2015) or coincident Cloud-Aerosol Lidar with Orthogonal Polarization (CALIOP) observations (Heidinger et al, 2016(Heidinger et al, , 2012 or from a scientific product (Merchant et al, 2005).…”

Leveraging spatial textures, through machine learning, to identify aerosol and distinct cloud types from multispectral observations

“…a million labeled images) to estimate the parameters of the CNN (i.e. the convolutional filters and bias terms), since the number of parameters required to accurately 275 identify different image types increases exponentially with the size of the input images (Gilton et al, 2020). In this paper the training dataset is relatively small since it is time-consuming to hand-label MODIS/VIIRS images and for optically thick aerosol there are not enough events in the observations.…”