Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions

Abstract

We propose a new type of neural networks, Kronecker neural networks (KNNs), that form a general framework for neural networks with adaptive activation functions. KNNs employ the Kronecker product, which provides an efficient way of constructing a very wide network while keeping the number of parameters low. Our theoretical analysis reveals that under suitable conditions, KNNs induce a faster decay of the loss than that by the feed-forward networks. This is also empirically verified through a set of computational examples. Furthermore, under certain technical assumptions, we establish global convergence of gradient descent for KNNs. As a specific case, we propose the Rowdy activation function that is designed to get rid of any saturation region by injecting sinusoidal fluctuations, which include trainable parameters. The proposed Rowdy activation function can be employed in any neural network architecture like feed-forward neural networks, Recurrent neural networks, Convolutional neural networks etc. The effectiveness of KNNs with Rowdy activation is demonstrated through various computational experiments including function approximation using feed-forward neural networks, solution inference of partial differential equations using the physicsinformed neural networks, and standard deep learning benchmark problems using convolutional and fully-connected neural networks.

Weight Matrix Dimensionality Reduction in Deep Learning via Kronecker Multi-layer Architectures

“…The Kronecker product has already been incorporated in several areas within the deep learning framework: (i) In [14,19] the authors apply a Kronecker product decomposition (KPD) to decompose weight matrices of a trained network, although this typically requires a large number of terms for acceptable accuracy and is thus of limited applicability; (ii) a generalized KPD is extended to multi-dimensional tensors in [8] to reduce the number of weight parameters and computational complexity in convolutional neural networks; (iii) the Kronecker product has been shown as a viable method to reduce the computational time for back-propagation via an approximate inverse of the Fisher information matrix, [13], providing a means to increase decay rate in the loss; (iv) a "Kronecker neural network" in [10], has been established to implement adaptive activation functions in order to avoid local minima while training. We emphasize that our approach is distinct from these methods, as we fundamentally alter network architecture in an attempt to accelerate training.…”

“…We now discuss the computational cost of a KDL-NN and give a broad technical explanation of why we expect the KDL-NN to be more efficient in practice. Given a KDL-NN defined by (10), gradient descent updates are performed on layer from L to 2 via the relations,…”