Dynamic fine‐tuning layer selection using <scp>Kullback</scp>–<scp>Leibler</scp> divergence

Wanjiku, Raphael Ngigi; Nderu, Lawrence; Kimwele, Michael

doi:10.1002/eng2.12595

Cited by 3 publications

(2 citation statements)

References 62 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This proposed work builds on two previous works by Wanjiku, Nderu & Kimwele (2022) . The first work looks into selecting relevant data points using textural features.…”

Section: Related Workmentioning

confidence: 86%

Improved transfer learning using textural features conflation and dynamically fine-tuned layers

Wanjiku,

Nderu,

Kimwele

2023

PeerJ Computer Science

Self Cite

View full text Add to dashboard Cite

Transfer learning involves using previously learnt knowledge of a model task in addressing another task. However, this process works well when the tasks are closely related. It is, therefore, important to select data points that are closely relevant to the previous task and fine-tune the suitable pre-trained model’s layers for effective transfer. This work utilises the least divergent textural features of the target datasets and pre-trained model’s layers, minimising the lost knowledge during the transfer learning process. This study extends previous works on selecting data points with good textural features and dynamically selected layers using divergence measures by combining them into one model pipeline. Five pre-trained models are used: ResNet50, DenseNet169, InceptionV3, VGG16 and MobileNetV2 on nine datasets: CIFAR-10, CIFAR-100, MNIST, Fashion-MNIST, Stanford Dogs, Caltech 256, ISIC 2016, ChestX-ray8 and MIT Indoor Scenes. Experimental results show that data points with lower textural feature divergence and layers with more positive weights give better accuracy than other data points and layers. The data points with lower divergence give an average improvement of 3.54% to 6.75%, while the layers improve by 2.42% to 13.04% for the CIFAR-100 dataset. Combining the two methods gives an extra accuracy improvement of 1.56%. This combined approach shows that data points with lower divergence from the source dataset samples can lead to a better adaptation for the target task. The results also demonstrate that selecting layers with more positive weights reduces instances of trial and error in selecting fine-tuning layers for pre-trained models.

show abstract

“…This proposed work builds on two previous works by Wanjiku, Nderu & Kimwele (2022) . The first work looks into selecting relevant data points using textural features.…”

Section: Related Workmentioning

confidence: 86%

Improved transfer learning using textural features conflation and dynamically fine-tuned layers

Wanjiku,

Nderu,

Kimwele

2023

PeerJ Computer Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…After selecting the best block, it is fully fine-tuned. "DKL" [59] uses the Kullback-Leibler divergence on weight correlations to identify the best layers for fine-tuning. Some techniques use an additional policy network for routing prediction: "Spot-Tune" [23] is based on the view that ResNet is an ensemble of shallow networks [69].…”

Section: Layer Routingmentioning

confidence: 99%

A Systematic Comparison of Task Adaptation Techniques for Digital Histopathology

Sauter,

Lodde,

Nensa

et al. 2023

Bioengineering

View full text Add to dashboard Cite

Due to an insufficient amount of image annotation, artificial intelligence in computational histopathology usually relies on fine-tuning pre-trained neural networks. While vanilla fine-tuning has shown to be effective, research on computer vision has recently proposed improved algorithms, promising better accuracy. While initial studies have demonstrated the benefits of these algorithms for medical AI, in particular for radiology, there is no empirical evidence for improved accuracy in histopathology. Therefore, based on the ConvNeXt architecture, our study performs a systematic comparison of nine task adaptation techniques, namely, DELTA, L2-SP, MARS-PGM, Bi-Tuning, BSS, MultiTune, SpotTune, Co-Tuning, and vanilla fine-tuning, on five histopathological classification tasks using eight datasets. The results are based on external testing and statistical validation and reveal a multifaceted picture: some techniques are better suited for histopathology than others, but depending on the classification task, a significant relative improvement in accuracy was observed for five advanced task adaptation techniques over the control method, i.e., vanilla fine-tuning (e.g., Co-Tuning: P(≫) = 0.942, d = 2.623). Furthermore, we studied the classification accuracy for three of the nine methods with respect to the training set size (e.g., Co-Tuning: P(≫) = 0.951, γ = 0.748). Overall, our results show that the performance of advanced task adaptation techniques in histopathology is affected by influencing factors such as the specific classification task or the size of the training dataset.

show abstract

An Adaptive Deep Learning Method Integrating Transfer Learning and Concept Drift Detection for Wind Power Forecasting

Wang,

Cao,

Jin

et al. 2024

2024 43rd Chinese Control Conference (CCC)

View full text Add to dashboard Cite

Dynamic fine‐tuning layer selection using Kullback–Leibler divergence

Cited by 3 publications

References 62 publications

Improved transfer learning using textural features conflation and dynamically fine-tuned layers

Improved transfer learning using textural features conflation and dynamically fine-tuned layers

A Systematic Comparison of Task Adaptation Techniques for Digital Histopathology

An Adaptive Deep Learning Method Integrating Transfer Learning and Concept Drift Detection for Wind Power Forecasting

Contact Info

Product

Resources

About