Dy-KD: Dynamic Knowledge Distillation for Reduced Easy Examples

Lin, Cheng; Jiang, Ning; Tang, Jialiang; Huang, Xinlei; Wu, Wenqing

doi:10.1007/978-981-99-8148-9_18

Cited by 2 publications

(1 citation statement)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There exists extensive literature studying knowledge distillation (Jiao et al 2020;Wang et al 2020;Gou et al 2021;Wu et al 2022;Ren et al 2023;Ji et al 2023;Li et al 2023). DeiT (Touvron et al 2021) introduces a distillation token to allow the vision transformer to learn from a ConvNet teacher.…”

Section: Knowledge Distillationmentioning

confidence: 99%

Transformer as Linear Expansion of Learngene

Xia,

Zhang,

Yang

et al. 2024

AAAI

View full text Add to dashboard Cite

We propose expanding the shared Transformer module to produce and initialize Transformers of varying depths, enabling adaptation to diverse resource constraints. Drawing an analogy to genetic expansibility, we term such module as learngene. To identify the expansion mechanism, we delve into the relationship between the layer's position and its corresponding weight value, and find that linear function appropriately approximates this relationship. Building on this insight, we present Transformer as Linear Expansion of learnGene (TLEG), a novel approach for flexibly producing and initializing Transformers of diverse depths. Specifically, to learn learngene, we firstly construct an auxiliary Transformer linearly expanded from learngene, after which we train it through employing soft distillation. Subsequently, we can produce and initialize Transformers of varying depths via linearly expanding the well-trained learngene, thereby supporting diverse downstream scenarios. Extensive experiments on ImageNet-1K demonstrate that TLEG achieves comparable or better performance in contrast to many individual models trained from scratch, while reducing around 2× training cost. When transferring to several downstream classification datasets, TLEG surpasses existing initialization methods by a large margin (e.g., +6.87% on iNat 2019 and +7.66% on CIFAR-100). Under the situation where we need to produce models of varying depths adapting for different resource constraints, TLEG achieves comparable results while reducing around 19× parameters stored to initialize these models and around 5× pre-training costs, in contrast to the pre-training and fine-tuning approach. When transferring a fixed set of parameters to initialize different models, TLEG presents better flexibility and competitive performance while reducing around 2.9× parameters stored to initialize, compared to the pre-training approach.

show abstract