Knowledge Transfer in Deep Convolutional Neural Nets

Gutstein, Steven; Fuentes, Olac; Freudenthal, Eric

doi:10.1142/s0218213008004059

Cited by 37 publications

(16 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The concept of learning from related tasks using neural networks and ConvNets has appeared earlier in the literature see [28,3,14,21] for a few examples. We describe two very recent papers which are the most relevant to our findings in this paper.…”

Section: Related Workmentioning

confidence: 99%

From generic to specific deep representations for visual recognition

Azizpour

Razavian

Sullivan

et al. 2015

2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

341

304

View full text Add to dashboard Cite

Evidence is mounting that ConvNets are the best representation learning method for recognition. In the common scenario, a ConvNet is trained on a large labeled dataset and the feed-forward units activation, at a certain layer of the network, is used as a generic representation of an input image. Recent studies have shown this form of representation to be astoundingly effective for a wide range of recognition tasks. This paper thoroughly investigates the transferability of such representations w.r.t. several factors. It includes parameters for training the network such as its architecture and parameters of feature extraction. We further show that different visual recognition tasks can be categorically ordered based on their distance from the source task. We then show interesting results indicating a clear correlation between the performance of tasks and their distance from the source task conditioned on proposed factors. Furthermore, by optimizing these factors, we achieve stateof-the-art performances on 16 visual recognition tasks.

show abstract

Section: Related Workmentioning

confidence: 99%

From generic to specific deep representations for visual recognition

Azizpour

Razavian

Sullivan

et al. 2015

2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

341

304

View full text Add to dashboard Cite

show abstract

“…Different from domain adaptation, an essential requirement for successful knowledge transfer is that the source domain and the target domain should be closely related [38,39]. Transfer learning has received much attention Manuscript submitted to ACM recently and many approaches based on CNNs have been proposed in the computer vision community [2,16,35,58,59].…”

Section: Transfer Learningmentioning

confidence: 99%

Multifaceted Analysis of Fine-Tuning in a Deep Model for Visual Recognition

Herranz

Jiang

2020

ACM/IMS Trans. Data Sci.

View full text Add to dashboard Cite

In recent years, convolutional neural networks (CNNs) have achieved impressive performance for various visual recognition scenarios.CNNs trained on large labeled datasets can not only obtain significant performance on most challenging benchmarks but also provide powerful representations, which can be used to a wide range of other tasks. However, the requirement of massive amounts of data to train deep neural networks is a major drawback of these models, as the data available is usually limited or imbalanced. Fine-tuning (FT) is an effective way to transfer knowledge learned in a source dataset to a target task. In this paper, we introduce and systematically investigate several factors that influence the performance of fine-tuning for visual recognition. These factors include parameters for the retraining procedure (e.g., the initial learning rate of fine-tuning), the distribution of the source and target data (e.g., the number of categories in the source dataset, the distance between the source and target datasets) and so on. We quantitatively and qualitatively analyze these factors, evaluate their influence, and present many empirical observations. The results reveal insights into what fine-tuning changes CNN parameters and provide useful and evidence-backed intuitions about how to implement fine-tuning for computer vision tasks.

show abstract

“…Transfer learning: Transfer learning is a machine learning approach where data from one problem is used to increase performance in another problem. The approach has previously been used for neural networks and ConvNets, see [2,12,15,19] for a few examples. In [12], the authors trained a ConvNet, viewing it as consisting of two halves, an earlier half and a later half.…”

Section: Related Workmentioning

confidence: 99%

“…The approach has previously been used for neural networks and ConvNets, see [2,12,15,19] for a few examples. In [12], the authors trained a ConvNet, viewing it as consisting of two halves, an earlier half and a later half. Transfer of knowledge was achieved by keeping the early layers as-is, and training the later layers for a new task.…”

Section: Related Workmentioning

confidence: 99%

Wooden Knot Detection Using ConvNet Transfer Learning

Norlander

Grahn

Maki

2015

Lecture Notes in Computer Science

View full text Add to dashboard Cite

This paper presents a method of localizing wooden knots in images of oak boards using deep convolutional networks (ConvNets). In particular, we show that transfer learning from generic images works effectively with a limited amount of available data when training a classifier for this highly specialized problem domain. We compare our method with a previous commercially developed technique based on kernel SVM with local feature descriptors. Our method is found to improve the detection performance significantly: F1 score 0.750 ± 0.018 vs 0.695. Furthermore, we report some observations regarding the behavior of KLdivergence on the test set which is counter-intuitive in its relation to the accuracy of classification.

show abstract

Knowledge Transfer in Deep Convolutional Neural Nets

Cited by 37 publications

References 12 publications

From generic to specific deep representations for visual recognition

From generic to specific deep representations for visual recognition

Multifaceted Analysis of Fine-Tuning in a Deep Model for Visual Recognition

Wooden Knot Detection Using ConvNet Transfer Learning

Contact Info

Product

Resources

About