Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network

Suzuki, Taiji; Abe, Hiroshi; Nishimura, Tatsuo

doi:10.48550/arxiv.1909.11274

Cited by 2 publications

(3 citation statements)

References 36 publications

(47 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To close with a few pointers to the literature, as Lemma 3.2 is essentially a pruning bound, it is potentially of independent interest; see for instance the literature on lottery tickets and pruning (Frankle and Carbin, 2019;Frankle et al, 2020;Su et al, 2020). Secondly, there is already one generalization bound in the literature which exhibits spectral norms, due to Suzuki et al (2019); unfortunately, it also has an explicit dependence on network width.…”

Section: Direct Uniform Convergence Approach In Theorem 14mentioning

confidence: 99%

“…1. We can directly apply uniform convergence; for instance, this approach was followed by Suzuki et al (2019), and a more direct approach is followed here to prove Theorem 1.4. Unfortunately, it is unclear how this technique can avoid paying significantly for the high complexity of f .…”

Section: An Abstract Bound Via Data Augmentationmentioning

confidence: 99%

“…width which often appear (the former appears in Theorem 1.3 and Lemma 3.1). Another stable rank bound was developed by Suzuki et al (2019), but has an extra mild dependence on width.…”

Section: A Uniform-convergence Approach To Distillationmentioning

confidence: 99%

See 2 more Smart Citations

Generalization bounds via distillation

Hsu¹,

Ji²,

Telgarsky³

et al. 2021

Preprint

View full text Add to dashboard Cite

This paper theoretically investigates the following empirical phenomenon: given a highcomplexity network with poor generalization bounds, one can distill it into a network with nearly identical predictions but low complexity and vastly smaller generalization bounds. The main contribution is an analysis showing that the original network inherits this good generalization bound from its distillation, assuming the use of well-behaved data augmentation. This bound is presented both in an abstract and in a concrete form, the latter complemented by a reduction technique to handle modern computation graphs featuring convolutional layers, fully-connected layers, and skip connections, to name a few. To round out the story, a (looser) classical uniform convergence analysis of compression is also presented, as well as a variety of experiments on cifar10 and mnist demonstrating similar generalization performance between the original network and its distillation.

show abstract

Section: Direct Uniform Convergence Approach In Theorem 14mentioning

confidence: 99%

Section: An Abstract Bound Via Data Augmentationmentioning

confidence: 99%

See 1 more Smart Citation

Generalization bounds via distillation

Hsu¹,

Ji²,

Telgarsky³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

Decomposable-Net: Scalable Low-Rank Compression for Neural Networks

Yaguchi

Suzuki

Nitta

et al. 2021

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

Compressing DNNs is important for the real-world applications operating on resource-constrained devices. However, we typically observe drastic performance deterioration when changing model size after training is completed. Therefore, retraining is required to resume the performance of the compressed models suitable for different devices. In this paper, we propose Decomposable-Net (the network decomposable in any size), which allows flexible changes to model size without retraining. We decompose weight matrices in the DNNs via singular value decomposition and adjust ranks according to the target model size. Unlike the existing low-rank compression methods that specialize the model to a fixed size, we propose a novel backpropagation scheme that jointly minimizes losses for both of full- and low-rank networks. This enables not only to maintain the performance of a full-rank network {\it without retraining} but also to improve low-rank networks in multiple sizes. Additionally, we introduce a simple criterion for rank selection that effectively suppresses approximation error. In experiments on the ImageNet classification task, Decomposable-Net yields superior accuracy in a wide range of model sizes. In particular, Decomposable-Net achieves the top-1 accuracy of 73.2% with 0.27xMACs with ResNet-50, compared to Tucker decomposition (67.4% / 0.30x), Trained Rank Pruning (70.6% / 0.28x), and universally slimmable networks (71.4% / 0.26x).

show abstract

Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network

Cited by 2 publications

References 36 publications

Generalization bounds via distillation

Generalization bounds via distillation

Decomposable-Net: Scalable Low-Rank Compression for Neural Networks

Contact Info

Product

Resources

About