A New Framework to Train Autoencoders Through Non-Smooth Regularization

Amini, Sajjad; Ghaemmaghami, Shahrokh

doi:10.1109/tsp.2019.2899294

Cited by 12 publications

(31 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sparse feature learning [14]- [19], L+S regularization of network weights [20] (U-Net trained with L+S output loss)…”

Section: Neural-model-based Mathematical-model-basedmentioning

confidence: 99%

“…There is plenty of room to incorporate sparsity-inducing priors into training as knowledge injection. They have explicitly been utilized for sparsification of hidden unit outputs in autoencoder-based sparse feature learning [14]- [19] as well as for low-rank and/or sparse regularization of network weights [5], [20]. The nuclear norm has not been used as a loss function yet, even though it is backpropable via automatic differentiation of the singular value decomposition [21].…”

Section: None (But Crucial To Prevent Catastrophic Forgetting)mentioning

confidence: 99%

“…Because of the nonsmoothness of these norms, most of the prior work mentioned above compromise some suboptimal training results by gradient-based methods with or without smoothing the norms. Proximal mapping as proposed in [19] is essential for the optimization of a loss function with sparsity-inducing norms, since the optimal solution locates at a non-differentiable point in the solution space. On fine tuning of a sparse-model-based network, using the sparsity-inducing norms as loss functions would be crucial to prevent catastrophic forgetting.…”

Section: None (But Crucial To Prevent Catastrophic Forgetting)mentioning

confidence: 99%

See 2 more Smart Citations

Unsupervised Deep Learning by Injecting Low-Rank and Sparse Priors

Sakai

2021

Preprint

View full text Add to dashboard Cite

What if deep neural networks can learn from sparsity-inducing priors? When the networks are designed by combining layer modules (CNN, RNN, etc), engineers less exploit the inductive bias, i.e., existing well-known rules or prior knowledge, other than annotated training data sets. We focus on employing sparsity-inducing priors in deep learning to encourage the network to concisely capture the nature of high-dimensional data in an unsupervised way. In order to use non-differentiable sparsity-inducing norms as loss functions, we plug their proximal mappings into the automatic differentiation framework. We demonstrate unsupervised learning of U-Net for background subtraction using low-rank and sparse priors. The U-Net can learn moving objects in a training sequence without any annotation, and successfully detect the foreground objects in test sequences.

show abstract

“…Sparse feature learning [14]- [19], L+S regularization of network weights [20] (U-Net trained with L+S output loss)…”

Section: Neural-model-based Mathematical-model-basedmentioning

confidence: 99%

Section: None (But Crucial To Prevent Catastrophic Forgetting)mentioning

confidence: 99%

Section: None (But Crucial To Prevent Catastrophic Forgetting)mentioning

confidence: 99%

See 1 more Smart Citation

Unsupervised Deep Learning by Injecting Low-Rank and Sparse Priors

Sakai

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Different types of regularization have been proposed to enhance the efficiency of FFNNs. Introduction of sparsity regularizer has successfully improved the initialization quality of deep fully-connected neural networks [9], [10]. Lowcoherence regularization has also shown impressive results in improving CNNs accuracy [11], which was previously explored for dictionary learning [12].…”

Section: Introductionmentioning

confidence: 99%

“…Smooth mathematical regularizers are preferable in the sense that they meet smoothness constraint assumed by many optimization algorithms used to learn network parameters [6]. Recently, a new learning framework has been proposed which aims at eliminating the smoothness constraint for the regularizers [9]. In this framework, instead of smoothness, the regularizer must be proximable which makes many nonsmooth regularizers applicable to the problem of training FFNNs.…”

Section: Introductionmentioning

confidence: 99%

Non-Smooth Regularization: Improvement to Learning Framework Through Extrapolation

Amini

Soltanian

Sadeghi

et al. 2022

IEEE Trans. Signal Process.

Self Cite

View full text Add to dashboard Cite

Deep learning architectures employ various regularization terms to handle different types of priors. Non-smooth regularization terms have shown promising performance in the deep learning architectures and a learning framework has recently been proposed to train autoencoders with such regularization terms. While this framework efficiently manages the non-smooth term during training through proximal operators, it is limited to autoencoders and suffers from low convergence speed due to several optimization sub-problems that must be solved in a row. In this paper, we address these issues by extending the framework to general feed-forward neural networks and introducing variable extrapolation which can dramatically increase the convergence speed in each sub-problem. We show that the proposed update rules converge to a critical point of the objective function under mild conditions. To compare the resulting framework with the previously proposed one, we consider the problem of training sparse autoencoders and robustifying deep neural architectures against both targeted and untargeted attacks. Simulations show superior performance in both convergence speed and final objective function value.

show abstract

A Parametric Lossy Compression Techniques for Biosignals: A Review

Dasan¹,

Gnanaraj²

2022

Wireless Pers Commun

View full text Add to dashboard Cite

A New Framework to Train Autoencoders Through Non-Smooth Regularization

Cited by 12 publications

References 40 publications

Unsupervised Deep Learning by Injecting Low-Rank and Sparse Priors

Unsupervised Deep Learning by Injecting Low-Rank and Sparse Priors

Non-Smooth Regularization: Improvement to Learning Framework Through Extrapolation

A Parametric Lossy Compression Techniques for Biosignals: A Review

Contact Info

Product

Resources

About