Stable Tensor Neural Networks for Rapid Deep Learning

Abstract

We propose a tensor neural network (t-NN) framework that offers an exciting new paradigm for designing neural networks with multidimensional (tensor) data. Our network architecture is based on the t-product [16], an algebraic formulation to multiply tensors via circulant convolution. In this t-product algebra, we interpret tensors as t-linear operators analogous to matrices as linear operators, and hence our framework inherits mimetic matrix properties. To exemplify the elegant, matrix-mimetic algebraic structure of our t-NNs, we expand on recent work [8] which interprets deep neural networks as discretizations of non-linear differential equations and introduces stable neural networks which promote superior generalization. Motivated by this dynamic framework, we introduce a stable t-NN which facilitates more rapid learning because of its reduced, more powerful parameterization. Through our high-dimensional design, we create a more compact parameter space and extract multidimensional correlations otherwise latent in traditional algorithms. We further generalize our t-NN framework to a family of tensor-tensor products [14] which still induce a matrix-mimetic algebraic structure. Through numerical experiments on the MNIST [20] and CIFAR-10 [17] datasets, we demonstrate the more powerful parameterizations and improved generalizability of stable t-NNs.

Randomized Kaczmarz for Tensor Linear Systems

“…A major advantage of ELM is that learning the linear mapping of the hidden-layer outputs to the output layer is relatively simple and is independent of the activation functions used. Newman et al proposed a tensor neural network intended for tensor data [27]. Their proposed networks use tensor-tensor products and enable the use of more compact parameter spaces [28].…”

“…Randomized Kaczmarz is closely related to the popular optimization technique, stochastic gradient descent (SGD) [26]. Most related to this work is the tensor stochastic gradient descent that was recently implemented to train tensor neural networks under the t-product [27]. The focus of the aforementioned work is a tensor neural network framework for multidimensional data and does not delve into an algorithmic analysis of SGD under the t-product.…”

“…Initially motivated for tensor factorization, use of the t-product has become prominent in the tensor and signal processing community. Under the t-product, tensors enjoy a linear algebraic-like framework that has proved useful in applications such as dictionary learning [33,38], low-rank tensor completion [41,32,39,40], facial recognition [11], and neural networks [27,36]. The process of naively transforming high-order tensors into two dimensional arrays via a flattening or unfolding process is often referred to as "matricization".…”

Sketch-and-project methods for tensor linear systems

“…where A ∈ R m×n×l , X ∈ R n×p×l and B ∈ R m×p×l are third-order tensors, and the operator * denotes the T-product introduced by Kilmer and Martin [1]. The problem (1.1) arises in many applications, including tensor dictionary learning [2][3][4][5][6][7], tensor neural network [8], boundary finite element method [9][10][11], etc. For T-product, it has an advantage that it can reserve the information inherent in the flattening of a tensor and, with it, many properties of numerical linear algebra can be extend to third and high order tensors [12][13][14][15][16][17][18].…”

T-Jordan Canonical Form and T-Drazin Inverse Based on the T-Product

“…where 'bcirc(A)' is the block circulant matrix [19] generated by the F-square tensor A ∈ C n×n×p . The T-function is also proved to be useful in stable tensor neural networks for rapid deep learning [31]. Special kinds of T-function such as tensor power has been used by Gleich, Chen and Varah [10] in Arnoldi methods to compute the eigenvalues of tensors and diagonal tensor canonical form was also proposed by them.…”