Manifold regularized deep neural networks

Tomar, Vikrant Singh; Rose, Richard C.

doi:10.21437/interspeech.2014-82

Cited by 17 publications

(9 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As a specific case, consider the use of manifold regularization in deep networks [44]. The basic idea is to force the NN to provide similar outputs whenever two inputs are 'close' according to some distance measure.…”

Section: E Case 4: Non-convex Regularizersmentioning

confidence: 99%

“…To this end, suppose that q ij is a non-negative value measuring the distance between the inputs x i and x j . Typically, this is defined as some measure of the Euclidean distance for the k-nearest neighbors of x i , and 0 otherwise [44]. A manifold regularization term can then be written as:…”

Section: E Case 4: Non-convex Regularizersmentioning

confidence: 99%

“…Note that, for a mini-batch of elements, computation of ( 26) requires Lk additional forward/backward computations in general [44], one for each neighbor of the elements in the mini-batch. We can handle this sort of regularization by replacing r(•) in ( 4) with a strongly convex approximation, following the same methodology as in Section IV-A.…”

Section: E Case 4: Non-convex Regularizersmentioning

confidence: 99%

See 2 more Smart Citations

Stochastic Training of Neural Networks via Successive Convex Approximations

Scardapane

Lorenzo

2018

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

This paper proposes a new family of algorithms for training neural networks (NNs). These are based on recent developments in the field of nonconvex optimization, going under the general name of successive convex approximation techniques. The basic idea is to iteratively replace the original (nonconvex, highly dimensional) learning problem with a sequence of (strongly convex) approximations, which are both accurate and simple to optimize. Different from similar ideas (e.g., quasi-Newton algorithms), the approximations can be constructed using only first-order information of the NN function, in a stochastic fashion, while exploiting the overall structure of the learning problem for a faster convergence. We discuss several use cases, based on different choices for the loss function (e.g., squared loss and cross-entropy loss), and for the regularization of the NN's weights. We experiment on several medium-sized benchmark problems and on a large-scale data set involving simulated physical data. The results show how the algorithm outperforms the state-of-the-art techniques, providing faster convergence to a better minimum. Additionally, we show how the algorithm can be easily parallelized over multiple computational units without hindering its performance. In particular, each computational unit can optimize a tailored surrogate function defined on a randomly assigned subset of the input variables, whose dimension can be selected depending entirely on the available computational power.

show abstract

Section: E Case 4: Non-convex Regularizersmentioning

confidence: 99%

Section: E Case 4: Non-convex Regularizersmentioning

confidence: 99%

Section: E Case 4: Non-convex Regularizersmentioning

confidence: 99%

See 1 more Smart Citation

Stochastic Training of Neural Networks via Successive Convex Approximations

Scardapane

Lorenzo

2018

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

show abstract

“…Two sets D sim and D dis denote the sets of data pairs with the same labels and different labels, respectively. Another form is defined in Tomar & Rose (2014) as follows: Lu et al (2015), Monperrus (2004), andRifai et al (2011)] are also possible; however, variations still have limitations inherited from the original manifold learning. Training set neighborhood information may be problematic because most nearest samples have too little in common in high-dimensional Euclidean spaces (Bengio et al, 2013).…”

Section: Deep Manifold Learningmentioning

confidence: 99%

“…1(b)). There have been many attempts (Reed et al, 2014;Tomar & Rose, 2014;Yuan et al, 2015) to unify deep learning and manifold learning. These studies have common limitations inherited from those of the original manifold learning.…”

Section: Introductionmentioning

confidence: 99%

Manifold Regularized Deep Neural Networks using Adversarial Examples

Lee,

Choi,

Yoon

2015

Preprint

View full text Add to dashboard Cite

Learning meaningful representations using deep neural networks involves designing efficient training schemes and well-structured networks. Currently, the method of stochastic gradient descent that has a momentum with dropout is one of the most popular training protocols. Based on that, more advanced methods (i.e., Maxout and Batch Normalization) have been proposed in recent years, but most still suffer from performance degradation caused by small perturbations, also known as adversarial examples. To address this issue, we propose manifold regularized networks (MRnet) that utilize a novel training objective function that minimizes the difference between multi-layer embedding results of samples and those adversarial. Our experimental results demonstrated that MRnet is more resilient to adversarial examples and helps us to generalize representations on manifolds. Furthermore, combining MRnet and dropout allowed us to achieve competitive classification performances for three well-known benchmarks: MNIST, CIFAR-10, and SVHN.

show abstract