Information Theoretic Representation Distillation

Miles, Roy; Rodriguez, Adrian Lopez; Mikolajczyk, Krystian

doi:10.48550/arxiv.2112.00459

Cited by 5 publications

(7 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Through the quantification of visual concepts encoded in teacher networks, Cheng et al [96] explain the success of knowledge distillation from the following three hypotheses: learning more visual concepts, learning various visual concepts, and yielding more stable optimization directions. Miles et al [97] integrate the analysis of information theory with knowledge distillation using infinitely divisible kernels, which achieve the computationally-efficient learning process on the crossmodel transfer tasks. Therefore, the quantification of knowledge can be investigated in the future research direction, which aims to analyze how much important knowledge can be potentially captured before the knowledge-learning process.…”

Section: Quality Of Knowledgementioning

confidence: 99%

Teacher-Student Architecture for Knowledge Learning: A Survey

Hu¹,

Li²,

Liú³

et al. 2022

Preprint

View full text Add to dashboard Cite

Although Deep Neural Networks (DNNs) have shown a strong capacity to solve large-scale problems in many areas, such DNNs with voluminous parameters are hard to be deployed in a real-time system. To tackle this issue, Teacher-Student architectures were first utilized in knowledge distillation, where simple student networks can achieve comparable performance to deep teacher networks. Recently, Teacher-Student architectures have been effectively and widely embraced on various knowledge learning objectives, including knowledge distillation, knowledge expansion, knowledge adaption, and multi-task learning. With the help of Teacher-Student architectures, current studies are able to achieve multiple knowledge-learning objectives through lightweight and effective student networks. Different from the existing knowledge distillation surveys, this survey detailedly discusses Teacher-Student architectures with multiple knowledge learning objectives. In addition, we systematically introduce the knowledge construction and optimization process during the knowledge learning and then analyze various Teacher-Student architectures and effective learning schemes that have been leveraged to learn representative and robust knowledge. This paper also summarizes the latest applications of Teacher-Student architectures based on different purposes (i.e., classification, recognition, and generation). Finally, the potential research directions of knowledge learning are investigated on the Teacher-Student architecture design, the quality of knowledge, and the theoretical studies of regression-based learning, respectively. With this comprehensive survey, both industry practitioners and the academic community can learn insightful guidelines about Teacher-Student architectures on multiple knowledge learning objectives.

show abstract

Section: Quality Of Knowledgementioning

confidence: 99%

Teacher-Student Architecture for Knowledge Learning: A Survey

Hu¹,

Li²,

Liú³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…(15) follows by noticing that Tn (x) ∈ [−1, 1] for any x ∈ [0, v]. (16) follows by applying Lemma A.3 on R(i−α, 2α+1) similar to (13). (17) follows by Euler's reflection formula similar to (14).…”

Section: Tr(pmentioning

confidence: 99%

“…Inspired by the quantum generalization of Rényi's definition [8], this new family of information measures is defined on the eigenspectrum of a normalized Hermitian matrix constructed by projecting data points in reproducing kernel Hilbert space (RKHS), thus avoiding explicit estimation of underlying data distributions. Because of its intriguing property in high-dimensional scenarios, the matrix-based Rényi's entropy and mutual information have been successfully applied in various data science applications, ranging from classical dimensionality reduction [9], [10] and feature selection [11] problems to advanced deep learning problems such as robust learning against covariant shift [5], network pruning [12] and knowledge distillation [13].…”

Section: Introductionmentioning

confidence: 99%

Optimal Randomized Approximations for Matrix based Renyi's Entropy

Yan¹,

Gong²,

Yu³

et al. 2022

Preprint

View full text Add to dashboard Cite

The Matrix-based Rényi's entropy enables us to directly measure information quantities from given data without the costly probability density estimation of underlying distributions, thus has been widely adopted in numerous statistical learning and inference tasks. However, exactly calculating this new information quantity requires access to the eigenspectrum of a semi-positive definite (SPD) matrix A which grows linearly with the number of samples n, resulting in a O(n 3 ) time complexity that is prohibitive for large-scale applications. To address this issue, this paper takes advantage of stochastic trace approximations for matrix-based Rényi's entropy with arbitrary α ∈ R + orders, lowering the complexity by converting the entropy approximation to a matrix-vector multiplication problem. Specifically, we develop random approximations for integerorder α cases and polynomial series approximations (Taylor and Chebyshev) for non-integer α cases, leading to a O(n 2 sm) overall time complexity, where s, m n denote the number of vector queries and the polynomial order respectively. We theoretically establish statistical guarantees for all approximation algorithms and give explicit order of s and m with respect to the approximation error , showing optimal convergence rate for both parameters up to a logarithmic factor. Large-scale simulations and real-world applications validate the effectiveness of the developed approximations, demonstrating remarkable speedup with negligible loss in accuracy.

show abstract

“…Inspired by the quantum generalization of Rényi's definition [8], this new family of information measures is defined on the eigenspectrum of a normalized Hermitian matrix constructed by projecting data points in reproducing kernel Hilbert space (RKHS), thus avoiding explicit estimation of underlying data distributions. Because of its intriguing property in high-dimensional scenarios, the matrix-based Rényi's entropy, and mutual information have been successfully applied in various data science applications, ranging from classical dimensionality reduction [9] and feature selection [10] problems to advanced deep learning problems such as robust learning against covariant shift [5], network pruning [11] and knowledge distillation [12].…”

Section: Introductionmentioning

confidence: 99%

Optimal Randomized Approximations for Matrix-Based Rényi’s Entropy

Dong¹,

Gong²,

et al. 2023

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal Take down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.

show abstract

Information Theoretic Representation Distillation

Cited by 5 publications

References 32 publications

Teacher-Student Architecture for Knowledge Learning: A Survey

Teacher-Student Architecture for Knowledge Learning: A Survey

Optimal Randomized Approximations for Matrix based Renyi's Entropy

Optimal Randomized Approximations for Matrix-Based Rényi’s Entropy

Contact Info

Product

Resources

About