On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks

Sulam, Jeremias; Aberdam, Aviad; Beck, Amir; Elad, Michael

doi:10.1109/tpami.2019.2904255

Cited by 77 publications

(70 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The networks we are about to experiment with are obtained as unfoldings of the L-THR (Figure 4a) and the L-BP ( Figure 4b) pursuit algorithms, and each is trained in a supervised fashion using back-propagation for best classification performance. Our tested architectures are relatively simple and use a small number of parameters in order to isolate the effect of their differences [24,2].…”

Section: Real Data Experimentsmentioning

confidence: 99%

Adversarial Noise Attacks of Deep Learning Architectures: Stability Analysis via Sparse-Modeled Signals

et al. 2019

Self Cite

View full text Add to dashboard Cite

Despite their impressive performance, deep convolutional neural networks (CNN) have been shown to be sensitive to small adversarial perturbations. These nuisances, which one can barely notice, are powerful enough to fool sophisticated and well performing classifiers, leading to ridiculous misclassification results. In this paper we analyze the stability of state-of-the-art deep-learning classification machines to adversarial perturbations, where we assume that the signals belong to the (possibly multi-layer) sparse representation model. We start with convolutional sparsity and then proceed to its multi-layered version, which is tightly connected to CNN. Our analysis links between the stability of the classification to noise and the underlying structure of the signal, quantified by the sparsity of its representation under a fixed dictionary. In addition, we offer similar stability theorems for two practical pursuit algorithms, Y. Romano and A. Aberdam contributed equally to this work. which are posed as two different deep-learning architectures -the Layered Thresholding and the Layered Basis Pursuit. Our analysis establishes the better robustness of the later to adversarial attacks. We corroborate these theoretical results by numerical experiments on three datasets: MNIST, CIFAR-10 and CIFAR-100.

show abstract

Section: Real Data Experimentsmentioning

confidence: 99%

Adversarial Noise Attacks of Deep Learning Architectures: Stability Analysis via Sparse-Modeled Signals

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…The resulting CSC model, where a special circulant and convolutional structure is imposed on dictionaries (which are otherwise traditionally unstructured in sparse coding theory) is defined as a forward pass of CNN [39]. Further work in a multilayered version of CSC has been shown in [42] for convergence analysis and multi-layer basis pursuit for classification performance comparison with CNNs on three public datasets.…”

Section: A Solvers For Cs Mri Reconstruction Problemmentioning

confidence: 99%

“…The greedy pursuit algorithm presented in [43] does not scale for high dimensional settings. J.Sulam et al [42] presented multi-layer basis pursuit algorithm that could leverage the symbiotic relationship of analysis and synthesis-priors on sparse representations of different layers according to the depth of multi-layer basis pursuit framework. Specifically, a convex relation was proposed for (11), resulting in multilayer basis pursuit:…”

Section: Multi Layered Basis Pursuitmentioning

confidence: 99%

Multi-Layer Basis Pursuit for Compressed Sensing MR Image Reconstruction

et al. 2020

View full text Add to dashboard Cite

Compressive Sensing (CS) is a widely used technique in biomedical signal acquisition and reconstruction. The technique is especially useful for reducing acquisition time for magnetic resonance imaging (MRI) signal acquisitions and reconstruction, where effects of patient fatigue and Claustrophobia need mitigation. In addition to improving patient experience, faster MRI scans are important for time sensitive imaging, such as functional or cardiac MRI, where target movement is unavoidable. Inspired from recent research works on multi-layer convolutional sparse coding (ML-CSC) theory to model deep neural networks, this work proposes a multi-layer basis pursuit framework which combines the benefit from objective-based CS reconstructions and deep learning-based reconstruction by employing iterative thresholding algorithms for successfully training a CS-MRI restoration framework on GPU and reconstruct test images using parameters of the trained model. Extensive experiments show the effectiveness of the proposed framework on four MRI datasets in terms of faster convergence, improved PSNR/SSIM, and better restoration efficiency as compared to the state of the art frameworks with different CS ratios. INDEX TERMS Compressive sensing, inverse problems in imaging, iterative thresholding algorithms, multi-layered convolutional sparse coding

show abstract

“…Exploring how similar ideas can be implemented in terms of appropriate network architectures remains a promising and interesting open question. Indeed, some initial ideas have already appeared in the recent work [36], and we believe several others will follow.…”

mentioning

confidence: 90%

Multi-Layer Sparse Coding: The Holistic Way

Aberdam¹,

Sulam²,

Elad³

2019

SIAM Journal on Mathematics of Data Science

Self Cite

View full text Add to dashboard Cite

The recently proposed multi-layer sparse model has raised insightful connections between sparse representations and convolutional neural networks (CNN). In its original conception, this model was restricted to a cascade of convolutional synthesis representations. In this paper, we start by addressing a more general model, revealing interesting ties to fully connected networks. We then show that this multi-layer construction admits a brand new interpretation in a unique symbiosis between synthesis and analysis models: while the deepest layer indeed provides a synthesis representation, the mid-layers decompositions provide an analysis counterpart. This new perspective exposes the suboptimality of previously proposed pursuit approaches, as they do not fully leverage all the information comprised in the model constraints. Armed with this understanding, we address fundamental theoretical issues, revisiting previous analysis and expanding it. Motivated by the limitations of previous algorithms, we then propose an integrated -holistic -alternative that estimates all representations in the model simultaneously, and analyze all these different schemes under stochastic noise assumptions. Inspired by the synthesis-analysis duality, we further present a Holistic Pursuit algorithm, which alternates between synthesis and analysis sparse coding steps, eventually solving for the entire model as a whole, with provable improved performance. Finally, we present numerical results that demonstrate the practical advantages of our approach.

show abstract

On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks

Cited by 77 publications

References 41 publications

Adversarial Noise Attacks of Deep Learning Architectures: Stability Analysis via Sparse-Modeled Signals

Adversarial Noise Attacks of Deep Learning Architectures: Stability Analysis via Sparse-Modeled Signals

Multi-Layer Basis Pursuit for Compressed Sensing MR Image Reconstruction

Multi-Layer Sparse Coding: The Holistic Way

Contact Info

Product

Resources

About