Mind the Gap in Distilling StyleGANs

Abstract

StyleGAN family is one of the most popular Generative Adversarial Networks (GANs) for unconditional generation. Despite its impressive performance, its high demand on storage and computation impedes their deployment on resource-constrained devices. This paper provides a comprehensive study of distilling from the popular StyleGANlike architecture. Our key insight is that the main challenge of Style-GAN distillation lies in the output discrepancy issue, where the teacher and student model yield different outputs given the same input latent code. Standard knowledge distillation losses typically fail under this heterogeneous distillation scenario. We conduct thorough analysis about the reasons and effects of this discrepancy issue, and identify that the mapping network plays a vital role in determining semantic information of generated images. Based on this finding, we propose a novel initialization strategy for the student model, which can ensure the output consistency to the maximum extent. To further enhance the semantic consistency between the teacher and student model, we present a latentdirection-based distillation loss that preserves the semantic relations in latent space. Extensive experiments demonstrate the effectiveness of our approach in distilling StyleGAN2 and StyleGAN3, outperforming existing GAN distillation methods by a large margin. Code is available at: https://github.com/xuguodong03/StyleKD

Head3D: Complete 3D Head Generation via Tri-plane Feature Distillation

“…logits and features, from teacher networks to student network, which is originally used in classification [10,37,43,55,49,62]. Following, KD is also employed for GAN-based model compression [9,52,30,32,54,25]. In GAN compression [30], the student network learns every intermediate features and final outputs from the teacher network.…”

“…CAGAN [32] adopts multi-Level distillation and content-aware distillation, then fine-tuned by adversarial loss. StyleKD [54] proposes that the mapping network plays an important role in generation. In addition, a novel initialization strategy and a latent direction-based distillation loss are presented for semantic consistency between the teacher and student model.…”

“…In order to achieve photo-realistic rendering results, GAN loss is necessary in training the tri-plane distillation framework [54]. The importance of camera poses in learning correct 3D priors in the discriminator has been highlighted in EG3D [7].…”

“…Detailed Texture Learning. StyleKD [54] proposes that mapping network determines semantic informations of generators. Therefore, besides tri-plane feature distillation, it is also necessary to ensure that the output w of the mapping network is consistent.…”

“…Therefore, besides tri-plane feature distillation, it is also necessary to ensure that the output w of the mapping network is consistent. Following StyleKD [54], a mapping loss is utilized:…”