On regularization for a convolutional kernel in neural networks

Abstract

Convolutional neural network is an important model in deep learning. To avoid exploding/vanishing gradient problems and to improve the generalizability of a neural network, it is desirable to have a convolution operation that nearly preserves the norm, or to have the singular values of the transformation matrix corresponding to a convolutional kernel bounded around 1. We propose a penalty function that can be used in the optimization of a convolutional neural network to constrain the singular values of the transformation matrix around 1. We derive an algorithm to carry out the gradient descent minimization of this penalty function in terms of convolution kernels. Numerical examples are presented to demonstrate the effectiveness of the method.

SVMax: A Feature Embedding Regularizer

“…Network weight regularizers dominate the deep learning regularizer literature because they support a large spectrum of tasks and architectures. Singular value decomposition (SVD) has been applied as a weight regularizer in several recent works (Zhang et al, 2018;Sedghi et al, 2018;Guo & Ye, 2019). Zhang et al (2018) employ SVD to avoid vanishing and exploding gradients in recurrent neural networks.…”

“…Zhang et al (2018) employ SVD to avoid vanishing and exploding gradients in recurrent neural networks. Similarly, Guo & Ye (2019) bound the singular values of the convolutional layer around 1 to preserve the layer's input and output norms. A bounded output norm mitigates the exploding/vanishing gradient problem.…”