On the regularization of convolutional kernel tensors in neural networks

A Frobenius norm regularization method for convolutional kernels to avoid unstable gradient problem

“…Figure 4.1: Convergence of σ max (M) and σ min (M) for different kernel sizes We would like to point out, we have used R 1 (K) = M T M − I 2 F to do numerical experiments on other random generated examples, including random kernels with each entry uniformly distributed on [0, 1]. The convergence figures of σ max (M) and σ min (M) are similar with the subfigures in Figure 4.1.We noticed that in[7] a 2-norm regularization method about convolutional kernels is proposed. The difference between 2-norm method in[7] and Frobenius norm method in this paper is needed to investigate further.…”

“…If the standard of constraining the singular values is not very high, one can stop the gradient descent process after first few steps. We noticed that in [7] a 2-norm regularization method about convolutional kernels is proposed. The difference between 2-norm method in [7] and Frobenius norm method in this paper is needed to investigate further.…”

“…We noticed that in [7] a 2-norm regularization method about convolutional kernels is proposed. The difference between 2-norm method in [7] and Frobenius norm method in this paper is needed to investigate further. Now we know that the computational cost at each iteration step is the updating of two singular vectors of M T M − αI for 2-norm method and the computation of the matrix M T M − αI for Frobenius norm method.…”

“…A drawback of the method in [12] is that projection can prevent the singular values of the transformation matrix being large but can't avoid the singular values to be too small. In [7] a 2-norm regularization method is proposed for convolutional kernels to constrain the singular values of the corresponding transformation matrices. In this paper we propose a Frobenius norm regularization method for convolutional kernels.…”

Regularization for convolutional kernel tensors to avoid unstable gradient problem in convolutional neural networks

“…In [7,8,17], regularization methods are proposed to let the corresponding transformation matrices be orthogonal, where the approach is to minimize the norm of M T M − I. In this paper we propose new regularization methods for the convolutional kernel tensor K, which can reduce the largest singular value and increase the smallest singular value of M independently or simultaneously depending on the need in the training process.…”