Spectral Pruning: Compressing Deep Neural Networks via Spectral Analysis and its Generalization Error

Abstract

The model size of deep neural network is getting larger and larger to realize superior performance in complicated tasks. This makes it difficult to implement deep neural network in small edge-computing devices. To overcome this problem, model compression methods have been gathering much attention. However, there have been only few theoretical back-grounds that explain what kind of quantity determines the compression ability. To resolve this issue, we develop a new theoretical frame-work for model compression, and propose a new method called Spectral-Pruning based on the theory. Our theoretical analysis is based on the observation such that the eigenvalues of the covariance matrix of the output from nodes in the internal layers often shows rapid decay. We define "degree of freedom" to quantify an intrinsic dimensionality of the model by using the eigenvalue distribution and show that the compression ability is essentially controlled by this quantity. Along with this, we give a generalization error bound of the compressed model. Our proposed method is applicable to wide range of models, unlike the existing methods, e.g., ones possess complicated branches as implemented in SegNet and ResNet. Our method makes use of both "input" and "output" in each layer and is easy to implement. We apply our method to several datasets to justify our theoretical analyses and show that the proposed method achieves the state-of-the-art performance.

Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network

“…The main difficulty in generalization error analysis of deep learning is that the Rademacher complexity of the full model F is quite large. One of the successful approaches to avoid this difficulty is the compression based bound [1,7,48] which measures how much the trained network f can be compressed. If the network can be compressed to much smaller one, then its intrinsic dimensionality can be regarded as small.…”

“…The refined version is given in Appendix A. This bound is general, and can be combined with the compression bounds derived so far such as [1,7,48] where the complexity of G and the bias r are analyzed for their generalization error bounds.…”

“…Here, we consider more data dependent bound: We assume the near low rank property of the covariance matrix among the nodes in an internal layer. A compression based bound for g using the low rank property of the covariance has been studied by [48], but their analysis requires a bit strong condition on the weight matrix. In this paper, we employ a weaker assumption.…”

“…Through these explicit and implicit regularizations, deep learning tends to produce a simpler model than its full expression ability [50,53]. To measure how "simple" the trained model is, one of the most promising approaches currently investigated is the compression bounds [1,7,48]. These bounds measure how much the network can be compressed and characterize the size of the compressed network as the implicit effective dimensionality.…”

“…Along with a similar direction, [7] proposed a pruning scheme called Corenet and derived a bound of the size of the compressed network. [48] has developed a spectrum based bound for their compression scheme. Unfortunately, all of these bounds guarantee the generalization error of only the compressed network, not the original network.…”