Ensemble Learning of Lightweight Deep Learning Models Using Knowledge Distillation for Image Classification

Kang, Jaeyong; Gwak, Jeonghwan

doi:10.3390/math8101652

Cited by 13 publications

(4 citation statements)

References 42 publications

(70 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…FitNet, the first feature-based method, is introduced to align intermediate representations layer by layer between the teacher and student models, aiming to enhance the student's performance. While this approach is simple and intuitive, it may face challenges related to convergence and performance due to the lack of high-level knowledge and the capacity gap between the two networks [28], [29]. A novel Exclusivity-Consistency regularized Knowledge Distillation (EC-KD) introduces a positionaware exclusivity strategy to enhance diversity among filters within the same layer, alleviate the limitations of student models and combine weight exclusivity and feature consistency in one unified framework [30].…”

Section: A Knowledge Distillationmentioning

confidence: 99%

Heterogeneous Student Knowledge Distillation From BERT Using a Lightweight Ensemble Framework

Lin,

Tsai,

Jwo

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Deep learning models have demonstrated their effectiveness in capturing complex relationships between input features and target outputs across many different application domains. These models, however, often come with considerable memory and computational demands, posing challenges for deployment on resource-constrained edge devices. Knowledge distillation is a prominent technique for transferring the expertise from an advanced yet heavy teacher model to a more efficient leaner student model. As ensemble methods have exhibited notable enhancements in model generalization and have achieved state-of-the-art performance in various machine learning tasks, we adopt ensemble techniques to perform knowledge distillation from BERT using multiple lightweight student models. Our approach applies lean architectural paradigms of spatial and sequential networks including LSTM, CNN and their fusion to perform data processing from distinct perspectives. Instead of using contextual word representations which require more space in natural language processing applications, we take advantage of a single static pre-trained and lowdimensional word embedding space to be shared among student models. Empirical studies are conducted on the sentiment classification problem and our model outperforms not only other existing techniques but also the teacher model.INDEX TERMS Knowledge distillation, ensemble methods, BERT, LSTM, CNN, contextual word representations, pre-trained and low-dimensional word embedding space, sentiment classification problem.

show abstract

Section: A Knowledge Distillationmentioning

confidence: 99%

Heterogeneous Student Knowledge Distillation From BERT Using a Lightweight Ensemble Framework

Lin,

Tsai,

Jwo

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…ResUNet and ResUNet?? have both been used effectively for polyp segmentation in medical image analysis [40][41][42][43]. Their ability to utilize skip connections and residual learning has enabled them to handle complex and diverse image datasets effectively.…”

Section: Deep Learning Modelsmentioning

confidence: 99%

PolyDSS: computer-aided decision support system for multiclass polyp segmentation and classification using deep learning

Saad,

Maghraby,

Badawy

2023

Neural Comput & Applic

View full text Add to dashboard Cite

Colorectal cancer (CRC) is a malignant condition that affects the colon or rectum, and it is distinguished by abnormal cell growth in these areas. Colon polyps, which are abnormalities, can turn into cancer. To stop the spread of cancer, early polyp detection is essential. The timely removal of polyps without submitting a sample for histology is made possible by computer-assisted polyp classification. In addition to Locally Shared Features (LSF) and ensemble learning majority voting, this paper introduces a computer-aided decision support system named PolyDSS to assist endoscopists in segmenting and classifying various polyp classes using deep learning models like ResUNet and ResUNet++ and transfer learning models like EfficientNet. The PICCOLO dataset is used to train and test the PolyDSS model. To address the issue of class imbalance, data augmentation techniques were used on the dataset. To investigate the impact of each technique on the model, extensive experiments were conducted. While the classification module achieved the highest accuracy of 0.9425 by utilizing the strength of ensemble learning using majority voting, the proposed segmenting module achieved the highest Dice Similarity Coefficient (DSC) of 0.9244 using ResUNet++ and LSF. In conjunction with the Paris classification system, the PolyDSS model, with its significant results, can assist clinicians in identifying polyps early and choosing the best approach to treatment.

show abstract

“…Deep ensemble learning models combine the advantages of deep learning and ensemble learning to improve the generalization performance of the model. In this regard, several researchers have used ensemble learning in their studies [41][42][43][44].…”

Section: Related Workmentioning

confidence: 99%

Ensemble of 2D Residual Neural Networks Integrated with Atrous Spatial Pyramid Pooling Module for Myocardium Segmentation of Left Ventricle Cardiac MRI

et al. 2022

View full text Add to dashboard Cite

Cardiac disease diagnosis and identification is problematic mostly by inaccurate segmentation of the cardiac left ventricle (LV). Besides, LV segmentation is challenging since it involves complex and variable cardiac structures in terms of components and the intricacy of time-based crescendos. In addition, full segmentation and quantification of the LV myocardium border is even more challenging because of different shapes and sizes of the myocardium border zone. The foremost purpose of this research is to design a precise automatic segmentation technique employing deep learning models for the myocardium border using cardiac magnetic resonance imaging (MRI). The ASPP module (Atrous Spatial Pyramid Pooling) was integrated with a proposed 2D-residual neural network for segmentation of the myocardium border using a cardiac MRI dataset. Further, the ensemble technique based on a majority voting ensemble method was used to blend the results of recent deep learning models on different set of hyperparameters. The proposed model produced an 85.43% dice score on validation samples and 98.23% on training samples and provided excellent performance compared to recent deep learning models. The myocardium border was successfully segmented across diverse subject slices with different shapes, sizes and contrast using the proposed deep learning ensemble models. The proposed model can be employed for automatic detection and segmentation of the myocardium border for precise quantification of reflow, myocardial infarction, myocarditis, and h cardiomyopathy (HCM) for clinical applications.

show abstract

Ensemble Learning of Lightweight Deep Learning Models Using Knowledge Distillation for Image Classification

Cited by 13 publications

References 42 publications

Heterogeneous Student Knowledge Distillation From BERT Using a Lightweight Ensemble Framework

Heterogeneous Student Knowledge Distillation From BERT Using a Lightweight Ensemble Framework

PolyDSS: computer-aided decision support system for multiclass polyp segmentation and classification using deep learning

Ensemble of 2D Residual Neural Networks Integrated with Atrous Spatial Pyramid Pooling Module for Myocardium Segmentation of Left Ventricle Cardiac MRI

Contact Info

Product

Resources

About