STKD: Distilling Knowledge From Synchronous Teaching for Efficient Model Compression

Su, Tongtong; Zhang, Jinsong; Yu, Zou; Wang, Gang; Liu, Xiaoguang

doi:10.1109/tnnls.2022.3164264

Cited by 11 publications

(2 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The binary codes method reduces the size of the model greatly and the computation on binary codes is much more faster than quantification methods. Sau and Balasubramanian [17] propose a knowledge distilling method in which the training of a compressed CNN model is taught by an original CNN model. Deep Compression Network [18] is a skillful integration among the compression methods mentioned above and it achieves the state‐of‐the‐art performance compared with original CNN models while its compression rate and speedup ratio is extremely high.…”

Section: Related Workmentioning

confidence: 99%

Beyond top‐N accuracy indicator: a comprehensive evaluation indicator of CNN models in image classification

Liu

Dou

Qiao

2020

IET Computer Vision

View full text Add to dashboard Cite

Nowadays, a large number of deep convolutional neural network (CNN) models are applied to image classification tasks. However, the authors find that the most widely used evaluation indicator, the Top‐N Accuracy indicator, cannot discriminate these models effectively. In this study, they propose a new indicator called Maximum‐Spanning‐Confusion‐Tree indicator to solve this problem. The Maximum‐Spanning‐Confusion‐Tree indicator is computed based on the hierarchical structure of the Maximum Spanning Confusion Tree of the deep CNN model on the dataset and reflect the ability of deep CNN models to discriminate confused categories in the dataset. The hierarchical structure of the Maximum Spanning Confusion Tree can reveal the confused category set of one selected category in the dataset efficiently and flexibly. Experiments show that they can discriminate ten different deep CNN models more accurately with the Maximum Spanning Confusion Tree indicator than the Top‐N Accuracy indicator and the Maximum Spanning Confusion Tree intuitively shows the distribution of confused category sets in the dataset so they can find out the weakness of deep CNN models effectively.

show abstract

Section: Related Workmentioning

confidence: 99%

Beyond top‐N accuracy indicator: a comprehensive evaluation indicator of CNN models in image classification

Liu

Dou

Qiao

2020

IET Computer Vision

View full text Add to dashboard Cite

show abstract

“…With the recent rapid development of deep neural network, 23 , 24 , 25 , 26 , 27 the combination of tomography and deep learning or machine learning can not only realize image analysis but also image reconstruction, which provides a new research direction for LDCT denoising. Due to the powerful feature extraction ability of the convolutional neural network (CNN), 28 it is increasingly applied to image denoising.…”

Section: Introductionmentioning

confidence: 99%

Self‐adaption and texture generation: A hybrid loss function for low‐dose CT denoising

Wang

Liu

Cheng

et al. 2023

J Applied Clin Med Phys

View full text Add to dashboard Cite

BackgroundDeep learning has been successfully applied to low‐dose CT (LDCT) denoising. But the training of the model is very dependent on an appropriate loss function. Existing denoising models often use per‐pixel loss, including mean abs error (MAE) and mean square error (MSE). This ignores the difference in denoising difficulty between different regions of the CT images and leads to the loss of large texture information in the generated image.PurposeIn this paper, we propose a new hybrid loss function that adapts to the noise in different regions of CT images to balance the denoising difficulty and preserve texture details, thus acquiring CT images with high‐quality diagnostic value using LDCT images, providing strong support for condition diagnosis.MethodsWe propose a hybrid loss function consisting of weighted patch loss (WPLoss) and high‐frequency information loss (HFLoss). To enhance the model's denoising ability of the local areas which are difficult to denoise, we improve the MAE to obtain WPLoss. After the generated image and the target image are divided into several patches, the loss weight of each patch is adaptively and dynamically adjusted according to its loss ratio. In addition, considering that texture details are contained in the high‐frequency information of the image, we use HFLoss to calculate the difference between CT images in the high‐frequency information part.ResultsOur hybrid loss function improves the denoising performance of several models in the experiment, and obtains a higher peak signal‐to‐noise ratio (PSNR) and structural similarity index (SSIM). Moreover, through visual inspection of the generated results of the comparison experiment, the proposed hybrid function can effectively suppress noise and retain image details.ConclusionsWe propose a hybrid loss function for LDCT image denoising, which has good interpretation properties and can improve the denoising performance of existing models. And the validation results of multiple models using different datasets show that it has good generalization ability. By using this loss function, high‐quality CT images with low radiation are achieved, which can avoid the hazards caused by radiation and ensure the disease diagnosis for patients.

show abstract

Lightweight fault diagnosis method in embedded system based on knowledge distillation

Gong,

Wang,

et al. 2023

J Mech Sci Technol

View full text Add to dashboard Cite

STKD: Distilling Knowledge From Synchronous Teaching for Efficient Model Compression

Cited by 11 publications

References 45 publications

Beyond top‐N accuracy indicator: a comprehensive evaluation indicator of CNN models in image classification

Beyond top‐N accuracy indicator: a comprehensive evaluation indicator of CNN models in image classification

Self‐adaption and texture generation: A hybrid loss function for low‐dose CT denoising

Lightweight fault diagnosis method in embedded system based on knowledge distillation

Contact Info

Product

Resources

About