Training of sparse and dense deep neural networks: Fewer parameters, same performance

Abstract

Working with high-dimensional data is a common practice, in the field of machine learning. Identifying relevant input features is thus crucial, so as to obtain compact dataset more prone for effective numerical handling. Further, by isolating pivotal elements that form the basis of decision making, one can contribute to elaborate on -ex post -models' interpretability, so far rather elusive. Here, we propose a novel method to estimate the relative importance of the input components for a Deep Neural Network. This is achieved by leveraging on a spectral re-parametrization of the optimization process. Eigenvalues associated to input nodes provide in fact a robust proxy to gauge the relevance of the supplied entry features. Unlike existing techniques, the spectral features ranking is carried out automatically, as a byproduct of the network training. The technique is successfully challenged against both synthetic and real data.

Spectral pruning of fully connected layers

“…This Section is devoted to reviewing the spectral approach to the training of deep neural networks. The discussion will follow mainly 6 , where an extension of the method originally introduced in 5 is handed over. For the sake of completeness, let us emphasize a substantial difference between these works 5 , 6 and the one proposed in this manuscript: in Giambagli et al 5 and Chicchi et al 6 the focus is on designing a training algorithm in the spectral domain while, in this work, we propose a novel idea to effectively prune fully connected layers by exploiting the spectral approach to neural network training.…”

“…Here, denotes the diagonal matrix of the eigenvalues of . Following 6 , we set for and . The remaining elements are initially assigned to random entries, as e.g.…”

“…To bridge the gap between conventional and spectral methods in terms of measured performances, one can also train the elements that populate the non trivial block of the eigenvectors matrix 5 . By resorting to apt decomposition schemes, it is still possible to contain the total number of trainable parameters, while reaching stunning performances in terms of classification outcomes 6 .…”

“…Alternatively, the learning can be carried out in reciprocal space: the spectral attributes (eigenvalues and eigenvectors) of the transfer operators that underlie information handling across layers define the actual target of the optimisation. This procedure, first introduced in 5 and further refined in 6 , enables a substantial compression of the space of trainable parameters. The spectral method leverages on a limited subset of key parameters which impact on the whole set of weights in direct space.…”

Recurrent Spectral Network (RSN): shaping the basin of attraction of a discrete map to reach automated classification

“…The target of the optimization are the weights of the links that connect pair of nodes belonging to adjacent stacks of the multilayered arrangement, in a fully coupled setting. An alternative training scheme has been recently proposed which anchors the learning to reciprocal domain [8,9]: the eigenvalues and the eigenvectors of the transfer operators get adjusted by the optimization. Spectral learning, so far engineered to deal with a feedforward organization, identifies key collective variables, the eigenvalues, which are more fundamental than any other (randomly selected) set of identical cardinality, allocated in direct space.…”