Deep Cross-Layer Collaborative Learning Network for Online Knowledge Distillation

Su, Tongtong; Liang, Qiyu; Zhang, Jinsong; Yu, Zou; Xu, Ziyue; Wang, Gang; Liu, Xiaoguang

doi:10.1109/tcsvt.2022.3222013

Cited by 7 publications

(10 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This design can improve the predictive power of compact models. [20]- [24] all design student-classmate ensemble training framework to obtain knowledge of ensemble teacher, which can guide both student and classmate efficiently in an end-to-end manner. AFID [65] directly employs one more complete sub-net to construct a two-branch ensemble training network.…”

Section: B Knowledge Distillation Guided Training Frameworkmentioning

confidence: 99%

“…In this paper, we select ResNet [20], [22]- [24], [51], [59]- [61], [63]. Specifically, KD-ONE [22], OEM [20], DCCL [24], AFID [65] and PCL [66] are recent notable methods which combines knowledge distillation strategy with ensemble learning. All these three methods are studentclassmate training framework (Fig.…”

Section: ) Classification On Lightweight Baseline Modelsmentioning

confidence: 99%

“…Naturally, if the performance of ensemble teacher is better, the main student is easier to become better. In Tab.X, we compare the performance of ensemble teacher with other ensemble knowledge distillation guided methods [20], [22], [24], [65] to reveal the all-round superiority of EKD-FWSNet. On CIFAR-10, EKD-FWSNet achieves comparable results with KD-ONE, where the error rate is a little (0.04%) higher.…”

Section: ) Classification On Lightweight Baseline Modelsmentioning

confidence: 99%

“…The top-left diagram indicates offline "teacher-student" training framework, where student model is optimized to approximate large-scale and mature teacher model. The top-right diagram indicates online "ensemble teacher-student" training framework [19]- [24], where studen-t model is part of a multi-branch ensemble network and gains knowledge on-the-fly from "ensemble teacher". This diagram focuses on utilizing the "ensemble teacher" to gain sub-optimal generalized knowledge due to the diversity and redundancy representation of multi-branch structure.…”

Section: Introductionmentioning

confidence: 99%

“…Previous ensemble training framework (Fig. 1 top-right) [19]- [24] constructs multiple forward branches by introducing multiple independent auxiliary classmate branches. Compared to these networks, EKD-FWSNet utilizes weight-sharing blocks to reduce the number and size of auxiliary classmate branches.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Learn by Oneself: Exploiting Weight-Sharing Potential in Knowledge Distillation Guided Ensemble Network

Zhao

Lyu

Chen

et al. 2023

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

Recent CNNs (convolutional neural networks) have become more and more compact. The elegant structure design highly improves the performance of CNNs. With the development of knowledge distillation technique, the performance of CNNs gets further improved. However, existing knowledge distillation guided methods either rely on offline pretrained high-quality large teacher models or online heavy training burden. To solve the above problems, we propose a feature-sharing and weightsharing based ensemble network (training framework) guided by knowledge distillation (EKD-FWSNet) to make baseline models stronger in terms of representation ability with less training computation and memory cost involved. Specifically, motivated by getting rid of the dependence of offline pretrained teacher model, we design an end-to-end online training scheme to optimize EKD-FWSNet. Motivated by decreasing the online training burden, we only introduce one auxiliary classmate branch to construct multiple forward branches, which will then be integrated as ensemble teacher to guide baseline model. Compared to previous online ensemble training frameworks, EKD-FWSNet can provide diverse output predictions without relying on increasing auxiliary classmate branches. Motivated by maximizing the optimization power of EKD-FWSNet, we exploit the representation potential of weight-sharing blocks and design efficient knowledge distillation mechanism in EKD-FWSNet. Extensive comparison experiments and visualization analysis on benchmark datasets (CIFAR-10/100, tiny-ImageNet, CUB-200 and ImageNet) show that self-learned EKD-FWSNet can boost the performance of baseline models by large margin, which has obvious superiority compared to previous related methods. Extensive analysis also proves the interpretability of EKD-FWSNet. Our code is available at https://github.com/cv516Buaa/EKD-FWSNet.

show abstract

Section: B Knowledge Distillation Guided Training Frameworkmentioning

confidence: 99%

Section: ) Classification On Lightweight Baseline Modelsmentioning

confidence: 99%

Section: ) Classification On Lightweight Baseline Modelsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Learn by Oneself: Exploiting Weight-Sharing Potential in Knowledge Distillation Guided Ensemble Network

Zhao

Lyu

Chen

et al. 2023

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

show abstract

Self‐adaption and texture generation: A hybrid loss function for low‐dose CT denoising

Wang

Liu

Cheng

et al. 2023

J Applied Clin Med Phys

View full text Add to dashboard Cite

BackgroundDeep learning has been successfully applied to low‐dose CT (LDCT) denoising. But the training of the model is very dependent on an appropriate loss function. Existing denoising models often use per‐pixel loss, including mean abs error (MAE) and mean square error (MSE). This ignores the difference in denoising difficulty between different regions of the CT images and leads to the loss of large texture information in the generated image.PurposeIn this paper, we propose a new hybrid loss function that adapts to the noise in different regions of CT images to balance the denoising difficulty and preserve texture details, thus acquiring CT images with high‐quality diagnostic value using LDCT images, providing strong support for condition diagnosis.MethodsWe propose a hybrid loss function consisting of weighted patch loss (WPLoss) and high‐frequency information loss (HFLoss). To enhance the model's denoising ability of the local areas which are difficult to denoise, we improve the MAE to obtain WPLoss. After the generated image and the target image are divided into several patches, the loss weight of each patch is adaptively and dynamically adjusted according to its loss ratio. In addition, considering that texture details are contained in the high‐frequency information of the image, we use HFLoss to calculate the difference between CT images in the high‐frequency information part.ResultsOur hybrid loss function improves the denoising performance of several models in the experiment, and obtains a higher peak signal‐to‐noise ratio (PSNR) and structural similarity index (SSIM). Moreover, through visual inspection of the generated results of the comparison experiment, the proposed hybrid function can effectively suppress noise and retain image details.ConclusionsWe propose a hybrid loss function for LDCT image denoising, which has good interpretation properties and can improve the denoising performance of existing models. And the validation results of multiple models using different datasets show that it has good generalization ability. By using this loss function, high‐quality CT images with low radiation are achieved, which can avoid the hazards caused by radiation and ensure the disease diagnosis for patients.

show abstract

Federated knowledge distillation for enhanced insulator defect detection in resource‐constrained environments

Huang,

Jia,

Tai

et al. 2024

IET Computer Vision

View full text Add to dashboard Cite

Insulator defect detection is crucial for the stable operation of power systems. It has become a mainstream research direction to realise insulator defect detection based on the combination of line images captured by UAVs and deep learning techniques. However, the existing high‐quality insulator defect detection models still face problems such as relying on massive‐labelled data and huge model parameters. Especially on resource‐constrained devices, it becomes a challenge to strike a balance between model lightweighting and performance. Although the knowledge distillation technique provides a solution for model lightweighting, the loss of information in the distillation process leads to the performance degradation of small models, which in turn creates a paradox between lightweighting and performance. Hence, an insulator defect detection method based on federated knowledge distillation is proposed. The method not only realises the lightweighting of the model, but also effectively improves the model performance by collaboratively training the model through the federated learning approach. Moreover, the asynchronous aggregation approach and model freshness mechanism designed in the method further enhance the training efficiency and collaborative effect. The experimental results show that the detection accuracy and efficiency of this paper's method on public datasets are significantly better than the benchmark algorithm.

show abstract

Deep Cross-Layer Collaborative Learning Network for Online Knowledge Distillation

Cited by 7 publications

References 52 publications

Learn by Oneself: Exploiting Weight-Sharing Potential in Knowledge Distillation Guided Ensemble Network

Learn by Oneself: Exploiting Weight-Sharing Potential in Knowledge Distillation Guided Ensemble Network

Self‐adaption and texture generation: A hybrid loss function for low‐dose CT denoising

Federated knowledge distillation for enhanced insulator defect detection in resource‐constrained environments

Contact Info

Product

Resources

About