Knowledge distillation in deep learning and its applications

Alkhulaifi, Abdolmaged; Alsahli, Fahad; Ahmad, Irfan

doi:10.7717/peerj-cs.474

Cited by 42 publications

(29 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The results indicate that a larger kernel of AlexNet (7×7) is more efficient on this task. In addition, to further optimize the detection system, we can deploy deep learning models on smartphones and get the result more conveniently and efficiently (Alkhulaifi et al 2021; Sujit et al 2021).…”

Section: Discussionmentioning

confidence: 99%

Detection of Frog virus 3 via the system integrating RPA-CRISPR/Cas12a-SPM with deep learning

Lei

Lian

Zhang

et al. 2022

Preprint

View full text Add to dashboard Cite

Frog virus 3 (FV3, genera Ranavirus, family Iridoviridae), a double-stranded DNA virus, results in irreparable damage to biodiversity and significant economic losses to aquaculture. Although the existing FV3 detection methods are of high sensitivity and specificity, the complex procedure and requirement of expensive instruments limit their practical implantation. Herein, we develop a fast, easy-to-implement, highly sensitive, and point-of-care (POC) detection system for FV3. Combining recombinase polymerase amplification (RPA) and CRISPR/Cas12a, we achieve a limit of detection (LoD) of 100 aM (60.2 copies/μL) by optimizing RPA primers and CRISPR RNAs (crRNAs). For POC detection, we build a smartphone microscopy (SPM) and achieve an LoD of 10 aM within 40 minutes. Four positive animal-derived samples with a quantitation cycle (Cq) value of quantitative PCR (qPCR) in the range of 13 to 32 are detectable by the proposed system. In addition, we deploy deep learning models for binary classification (positive or negative samples) and multiclass classification (different concentrations of FV3 and negative samples), achieving 100% and 98.75% accuracy, respectively. Without temperature regulation and expensive equipment, RPA-CRISPR/Cas12a combined with a smartphone readout and artificial intelligence (AI) assisted classification shows great potential for FV3 detection. This integrated system holds great promise for POC detection of aquatic DNA pathogens.

show abstract

Section: Discussionmentioning

confidence: 99%

Detection of Frog virus 3 via the system integrating RPA-CRISPR/Cas12a-SPM with deep learning

Lei

Lian

Zhang

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Teacher-Student architectures have been commonly applied in knowledge distillation for model compression, and some surveys [5], [6], [7] summarized the recent progress of various knowledge distillation techniques with Teacher-Student architectures. Specifically, Gou et al [5] presented a comprehensive survey on knowledge distillation from the following perspectives: knowledge types, distillation schemes, and Teacher-Student architectures.…”

Section: Introductionmentioning

confidence: 99%

“…Wang et al [6] provided a systematic overview and insight into knowledge distillation with Teacher-Student architectures in CV applications. Alkhulaifi et al [7] summarized multiple distillation metrics to compare the performances of different distillation methods. However, these aforementioned surveys do not discuss knowledge construction and optimization during the distillation process, where the knowledge types and optimization objectives are the important factors in providing informative knowledge for student learning.…”

Section: Introductionmentioning

confidence: 99%

“…Hence, this survey provides a comprehensive and insightful guideline about Teacher-Student architectures on knowledge learning. Different from the existing surveys on knowledge distillation [5], [6], [7], this paper first introduces Teacher-Student architectures on multiple knowledge learning objectives (including knowledge distillation, knowledge expansion, knowledge adaption, and multi-task learning), and then discusses the knowledge construction and optimization processes. Moreover, we systematically summarize various Teacher-Student architectures and learning schemes that can be utilized to learn representative and robust knowledge.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Teacher-Student Architecture for Knowledge Learning: A Survey

Hu¹,

Li²,

Liú³

et al. 2022

Preprint

View full text Add to dashboard Cite

Although Deep Neural Networks (DNNs) have shown a strong capacity to solve large-scale problems in many areas, such DNNs with voluminous parameters are hard to be deployed in a real-time system. To tackle this issue, Teacher-Student architectures were first utilized in knowledge distillation, where simple student networks can achieve comparable performance to deep teacher networks. Recently, Teacher-Student architectures have been effectively and widely embraced on various knowledge learning objectives, including knowledge distillation, knowledge expansion, knowledge adaption, and multi-task learning. With the help of Teacher-Student architectures, current studies are able to achieve multiple knowledge-learning objectives through lightweight and effective student networks. Different from the existing knowledge distillation surveys, this survey detailedly discusses Teacher-Student architectures with multiple knowledge learning objectives. In addition, we systematically introduce the knowledge construction and optimization process during the knowledge learning and then analyze various Teacher-Student architectures and effective learning schemes that have been leveraged to learn representative and robust knowledge. This paper also summarizes the latest applications of Teacher-Student architectures based on different purposes (i.e., classification, recognition, and generation). Finally, the potential research directions of knowledge learning are investigated on the Teacher-Student architecture design, the quality of knowledge, and the theoretical studies of regression-based learning, respectively. With this comprehensive survey, both industry practitioners and the academic community can learn insightful guidelines about Teacher-Student architectures on multiple knowledge learning objectives.

show abstract

“…Knowledge elements that are transferred to the student models can be output values of certain layers in the teacher network, for example, it may be logits that precede softmax in classification. It is also possible to use internal layer output values of the teacher network [2]. This method shows good results for training more compact networks while maintaining the required accuracy, but there is no standard approach for organizing such a process.…”

Section: Introductionmentioning

confidence: 99%

Local Feature Matching with Transformers for low-end devices

Kolodiazhnyi¹

2022

Preprint

View full text Add to dashboard Cite

LoFTR [19] is an efficient deep learning method for finding appropriate local feature matches on image pairs. This paper reports on the optimization of this method to work on devices with low computational performance and limited memory. The original LoFTR approach is based on a ResNet [6] head and two modules based on Linear Transformer [22] architecture. In the presented work, only the coarse-matching block was left, the number of parameters was significantly reduced, and the network was trained using a knowledge distillation technique. The comparison showed that this approach allows to obtain an appropriate feature detection accuracy for the student model compared to the teacher model in the coarse matching block, despite the significant reduction of model size. Also, the paper shows additional steps required to make model compatible with NVIDIA TensorRT runtime, and shows an approach to optimize training method for low-end GPUs.

show abstract

Knowledge distillation in deep learning and its applications

Cited by 42 publications

References 39 publications

Detection of Frog virus 3 via the system integrating RPA-CRISPR/Cas12a-SPM with deep learning

Detection of Frog virus 3 via the system integrating RPA-CRISPR/Cas12a-SPM with deep learning

Teacher-Student Architecture for Knowledge Learning: A Survey

Local Feature Matching with Transformers for low-end devices

Contact Info

Product

Resources

About