GANs Can Play Lottery Tickets Too

Chen, Xuxi; Zhang, Zhenyu; Sui, Yongduo; Chen, Tianlong

doi:10.48550/arxiv.2106.00134

Cited by 6 publications

(21 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, we conduct experiments to evaluate STU-GAN. Following [16], we choose the widely-used SNGAN [4] on CIFAR-10 for the image generation task. Moreover, to draw a more solid conclusion with large scale GANs, we also evaluate our method with BigGAN [6] trained on CIFAR-10 and ImageNet.…”

Section: Resultsmentioning

confidence: 99%

“…Moreover, to draw a more solid conclusion with large scale GANs, we also evaluate our method with BigGAN [6] trained on CIFAR-10 and ImageNet. To enable comparison among different methods, we follow [16] and employ two widely-used metrics Fréchet Inception Distance (FID) and Inception Score (IS) as the approximate measure of model performance.…”

Section: Resultsmentioning

confidence: 99%

“…A pre-trained discriminator is used by [22] to supervise the training of a compressed generator, achieving compelling results. Very recently, the authors of [16] SST represents a class of methods that aim to train a sparse subnetwork with a fixed sparse connectivity pattern during the course of training. The most naive approach is pruning each layer uniformly with the same pruning ratio, i.e., uniform pruning [28][29][30][31].…”

Section: Gan Compressionmentioning

confidence: 99%

“…Prior works utilize model compression techniques, such as pruning [10], distillation [11][12][13], quantization [14], and lottery tickets hypothesis (LTH) [15][16][17] to produce an efficient generator with competitive performance. While increasingly efficient, the existing techniques are not designed to accelerate training as they either operate on fully pre-trained GANs or require the over-parameterized dense GANS to be stored or updated during training.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Don't Be So Dense: Sparse-to-Sparse GAN Training Without Sacrificing Performance

Liu¹,

Tian²,

Chen³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Generative adversarial networks (GANs) have received an upsurging interest since being proposed due to the high quality of the generated data. While achieving increasingly impressive results, the resource demands associated with the large model size hinders the usage of GANs in resource-limited scenarios. For inference, the existing model compression techniques can reduce the model complexity with comparable performance. However, the training efficiency of GANs has less been explored due to the fragile training process of GANs. In this paper, we, for the first time, explore the possibility of directly training sparse GAN from scratch without involving any dense or pre-training steps. Even more unconventionally, our proposed method enables directly training sparse unbalanced GANs with an extremely sparse generator from scratch. Instead of training full GANs, we start with sparse GANs and dynamically explore the parameter space spanned over the generator throughout training. Such a sparse-to-sparse training procedure enhances the capacity of the highly sparse generator progressively while sticking to a fixed small parameter budget with appealing training and inference efficiency gains. Extensive experiments with modern GAN architectures validate the effectiveness of our method. Our sparsified GANs, trained from scratch in one single run, are able to outperform the ones learned by expensive iterative pruning and re-training. Perhaps most importantly, we find instead of inheriting parameters from expensive pre-trained GANs, directly training sparse GANs from scratch can be a much more efficient solution. For example, only training with a 80% sparse generator and a 70% sparse discriminator, our method can achieve even better performance than the dense BigGAN.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Gan Compressionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Don't Be So Dense: Sparse-to-Sparse GAN Training Without Sacrificing Performance

Liu¹,

Tian²,

Chen³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The lottery ticket hypothesis (LTH) has attracted widespread attention and has been evidenced in various traditional computer vision fields, such as image classification [13,42,43,51,55], object detection [20]. Recently, the properties of LTH has also been widely studied across other fields, such as natural language processing [5,16,47,57], reinforcement learning [57], graph neural networks [6], life-long learning [7], and generative adversarial networks [4,9,31]. On the other hand, the "rewinding late" rule is found by [15,49] to scale up LTH to larger networks and datasets.…”

Section: Lottery Ticket Hypothesismentioning

confidence: 99%

Playing Lottery Tickets in Style Transfer Models

Kong¹,

Huo²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

Style transfer has achieved great success and attracted a wide range of attention from both academic and industrial communities due to its flexible application scenarios. However, the dependence on pretty large VGG based autoencoder leads to existing style transfer models have a high parameter complexities which limits the application for resource-constrained devices. Unfortunately, the compression of style transfer model has less been explored. In parallel, study on the lottery ticket hypothesis (LTH) has shown great potential in finding extremely sparse matching subnetworks which can achieve on par or even better performance than original full networks when trained in isolation. In this work, we perform the first empirical study to verify whether such trainable networks also exist in style transfer models. From a wide range of style transfer methods, we choose two of the most popular style transfer models as the main testbeds, i.e., AdaIN and SANet, representing approaches of global and local transformation based style transfer respectively. Through extensive experiments and comprehensive analysis, we draw the following main conclusions. (𝑖) Compared with fixing VGG encoder, style transfer models can benefit more from training the whole network together. (𝑖𝑖) Using iterative magnitude pruning, we find the most sparse matching subnetworks at 89.2% in AdaIN and 73.7% in SANet, which suggests that Style transfer models can play lottery tickets too. (𝑖𝑖𝑖) Feature transformation module should also be pruned to get a sparser model without affecting the existence and quality of matching subnetworks. (𝑖𝑣) Besides AdaIN and SANet, other models such as LST, MANet, AdaAttN and MCCNet can also play lottert tickets, which shows that LTH can be generalized to various style transfer models. CCS CONCEPTS• Computing methodologies → Non-photorealistic rendering.

show abstract

Don’t Be So Dense: Sparse-to-Sparse GAN Training Without Sacrificing Performance

et al. 2023

View full text Add to dashboard Cite

published version features the final layout of the paper including the volume, issue and page numbers. Link to publication General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal.If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the "Taverne" license above, please follow below link for the End User

show abstract

GANs Can Play Lottery Tickets Too

Cited by 6 publications

References 17 publications

Don't Be So Dense: Sparse-to-Sparse GAN Training Without Sacrificing Performance

Don't Be So Dense: Sparse-to-Sparse GAN Training Without Sacrificing Performance

Playing Lottery Tickets in Style Transfer Models

Don’t Be So Dense: Sparse-to-Sparse GAN Training Without Sacrificing Performance

Contact Info

Product

Resources

About