ISTA-Net: Interpretable Optimization-Inspired Deep Network for Image Compressive Sensing

Zhang, Jian; Ghanem, Bernard

doi:10.1109/cvpr.2018.00196

Cited by 862 publications

(761 citation statements)

References 47 publications

Supporting

Mentioning

672

Contrasting

Order By: Relevance

“…Each layer has independent trainable weights and the whole network is trained under the supervision of ISTA solutions. In the related context of linear inverse problems, including compressed sensing [18,19,20,21] and image restoration [22,23,24], several works have followed this unrolling approach. In some cases, weight tying across layers is maintained as in the unrolled algorithm [22].…”

Section: Related Workmentioning

confidence: 99%

A Flexible Convolutional Solver for Fast Style Transfers

Puy

Pérez

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

1) Artistic style (2) Photorealistic style (a) Input image (b) Artistic transfer (c) Photorealistic transfer Figure 1. Given an input image (a), the proposed convolutional neural network yields fast artistic stylisation results, e.g., (b) for style (1), and can be modified at runtime -hence without retraining -to yield photorealistic stylisation results, e.g., (c) for style (2).Abstract. We propose a new flexible deep convolutional neural network (convnet) to perform fast visual style transfer. In contrast to existing convnets that address the same task, our architecture derives directly from the structure of the gradient descent originally used to solve the style transfer problem [1]. Like existing convnets, ours approximately solves the original problem much faster than the gradient descent. However, our network is uniquely flexible by design: it can be manipulated at runtime to enforce new constraints on the final solution. In particular, we show how to modify it to obtain a photorealistic result with no retraining. We study the modifications made by [2] to the original cost function of [1] to achieve photorealistic style transfer. These modifications affect directly the gradient descent and can be reported on-the-fly in our network. These modifications are possible as the proposed architecture stems from unrolling the gradient descent.Gilles Puy and Patrick Pérez are with Technicolor,

show abstract

Section: Related Workmentioning

confidence: 99%

A Flexible Convolutional Solver for Fast Style Transfers

Puy

Pérez

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…Classic optimization algorithms can be unfolded to perform many different tasks in image processing. For instance, FISTA and ISTA can be unfolded to perform sparse coding [22,23], while the same ISTA and ADMM can be unrolled for image compressive sensing [24,25]. However, in the aforementioned works, some functions and operators are learned, which weakens the link between the resulting network and the original algorithm.…”

Section: Introductionmentioning

confidence: 99%

Deep unfolding of a proximal interior point method for image restoration

Bertocchi¹,

Chouzenoux²,

Corbineau³

et al. 2020

Inverse Problems

105

View full text Add to dashboard Cite

Variational methods are widely applied to ill-posed inverse problems for they have the ability to embed prior knowledge about the solution. However, the level of performance of these methods significantly depends on a set of parameters, which can be estimated through computationally expensive and timeconsuming methods. In contrast, deep learning offers very generic and efficient architectures, at the expense of explainability, since it is often used as a black-box, without any fine control over its output. Deep unfolding provides a convenient approach to combine variational-based and deep learning approaches. Starting from a variational formulation for image restoration, we develop iRestNet, a neural network architecture obtained by unfolding a proximal interior point algorithm. Hard constraints, encoding desirable properties for the restored image, are incorporated into the network thanks to a logarithmic barrier, while the barrier parameter, the stepsize, and the penalization weight are learned by the network. We derive explicit expressions for the gradient of the proximity operator for various choices of constraints, which allows training iRestNet with gradient descent and backpropagation. In addition, we provide theoretical results regarding the stability of the network for a common inverse problem example. Numerical experiments on image deblurring problems show that the proposed approach compares favorably with both state-of-the-art variational and machine learning methods in terms of image quality.

show abstract

“…While other works have indeed explored the unrolling of iterative algorithms in terms of CNNs (e.g. [33], [44]), we are not aware of any work that has attempted nor studied the unrolling of a global pursuit with convergence guarantees. Lastly, we demonstrate the performance of these networks in practice by training our models for image classification, consistently improving on the classical feed-forward architectures without introducing filters nor any other extra parameters in the model.…”

Section: Introductionmentioning

confidence: 99%

On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks

Sulam

Aberdam

Beck

et al. 2020

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Parsimonious representations are ubiquitous in modeling and processing information. Motivated by the recent Multi-Layer Convolutional Sparse Coding (ML-CSC) model, we herein generalize the traditional Basis Pursuit problem to a multi-layer setting, introducing similar sparse enforcing penalties at different representation layers in a symbiotic relation between synthesis and analysis sparse priors. We explore different iterative methods to solve this new problem in practice, and we propose a new Multi-Layer Iterative Soft Thresholding Algorithm (ML-ISTA), as well as a fast version (ML-FISTA). We show that these nested first order algorithms converge, in the sense that the function value of near-fixed points can get arbitrarily close to the solution of the original problem. We further show how these algorithms effectively implement particular recurrent convolutional neural networks (CNNs) that generalize feed-forward ones without introducing any parameters. We present and analyze different architectures resulting unfolding the iterations of the proposed pursuit algorithms, including a new Learned ML-ISTA, providing a principled way to construct deep recurrent CNNs. Unlike other similar constructions, these architectures unfold a global pursuit holistically for the entire network. We demonstrate the emerging constructions in a supervised learning setting, consistently improving the performance of classical CNNs while maintaining the number of parameters constant.

show abstract

ISTA-Net: Interpretable Optimization-Inspired Deep Network for Image Compressive Sensing

Cited by 862 publications

References 47 publications

A Flexible Convolutional Solver for Fast Style Transfers

A Flexible Convolutional Solver for Fast Style Transfers

Deep unfolding of a proximal interior point method for image restoration

On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks

Contact Info

Product

Resources

About