Entropy and mutual information in models of deep neural networks*

Gabrié, Marylou; Manoel, Andre; Luneau, Clément; Barbier, Jean; Macris, Nicolas; Krząkała, Florent; Zdeborová, Lenka

doi:10.1088/1742-5468/ab3430

Cited by 102 publications

(111 citation statements)

References 42 publications

Supporting

Mentioning

108

Contrasting

Order By: Relevance

“…It is useful to compare the predicted MSE with the predicted optimal values. The works [47], [48] postulate the optimal MSE for inference in deep networks under the LSL model described above using the replica method from statistical physics. Interestingly, it is shown in [47,Thm.2] that the predicted minimum MSE satisfies equations that exactly agree with the fixed points of the updates (39).…”

Section: Mmse Estimation and Connections To The Replica Predictionsmentioning

confidence: 99%

Inference With Deep Generative Priors in High Dimensions

Pandit

Sahraee-Ardakan

Rangan

et al. 2020

IEEE J. Sel. Areas Inf. Theory

View full text Add to dashboard Cite

Deep generative priors offer powerful models for complex-structured data, such as images, audio, and text. Using these priors in inverse problems typically requires estimating the input and/or hidden signals in a multi-layer deep neural network from observation of its output. While these approaches have been successful in practice, rigorous performance analysis is complicated by the non-convex nature of the underlying optimization problems. This paper presents a novel algorithm, Multi-Layer Vector Approximate Message Passing (ML-VAMP), for inference in multi-layer stochastic neural networks. ML-VAMP can be configured to compute maximum a priori (MAP) or approximate minimum mean-squared error (MMSE) estimates for these networks. We show that the performance of ML-VAMP can be exactly predicted in a certain high-dimensional random limit. Furthermore, under certain conditions, ML-VAMP yields estimates that achieve the minimum (i.e., Bayes-optimal) MSE as predicted by the replica method. In this way, ML-VAMP provides a computationally efficient method for multi-layer inference with an exact performance characterization and testable conditions for optimality in the large-system limit.

show abstract

Section: Mmse Estimation and Connections To The Replica Predictionsmentioning

confidence: 99%

Inference With Deep Generative Priors in High Dimensions

Pandit

Sahraee-Ardakan

Rangan

et al. 2020

IEEE J. Sel. Areas Inf. Theory

View full text Add to dashboard Cite

show abstract

“…So far these include matrix and tensor factorisation [24], estimation in traditional and generalised linear models [25] (e.g., compressed sensing and many of its non-linear variants), with random i.i.d. as well as special structured measurement matrices [26], learning problems in the teacher-student setting [25,27] (e.g., the single-layer perceptron network) and even multi-layer versions [28]. For inference problems with an underlying sparse graphical structure full proofs of replica formulas are scarce and much more involved, e.g., [11,17].…”

mentioning

confidence: 99%

The adaptive interpolation method for proving replica formulas. Applications to the Curie–Weiss and Wigner spike models

Barbier¹,

Macris²

2019

J. Phys. A: Math. Theor.

Self Cite

View full text Add to dashboard Cite

In this contribution we give a pedagogic introduction to the newly introduced adaptive interpolation method to prove in a simple and unified way replica formulas for Bayesian optimal inference problems. Many aspects of this method can already be explained at the level of the simple Curie-Weiss spin system. This provides a new method of solution for this model which does not appear to be known. We then generalize this analysis to a paradigmatic inference problem, namely rank-one matrix estimation, also refered to as the Wigner spike model in statistics. We give many pointers to the recent literature where the method has been succesfully applied.Keywords adaptive interpolation · Bayesian inference · replica formula · matrix estimation · Wigner spike model · Curie-Weiss model · spin systems 1 IntroductionThe replica method from statistical mechanics has been applied to Bayesian inference problems (e.g., coding, estimation) already two decades ago [1,2]. Rigorous proofs of the formulas for the mutual informations/entropies/free energies stemming from this method, have for a long time only been partial, consisting generally of one sided bounds [3,4,5,6,7,8,9,10]. It is only quite recently that there has been a surge of progress using various methods -namely spatial coupling [11,12,13,14,15], information theory [16], and rigorous versions of the cavity method [17,18,19]-to derive full proofs, but which are typically quite complicated. Recently we introduced [20] a powerful evolution of the Guerra-Toninelli [21,22,23] interpolation method -called adaptive interpolation-that allows to fully prove the replica formulas in a quite simple and unified way for Bayesian inference problems. In this contribution we give a pedagogic introduction to this new method.We first illustrate how the adaptive interpolation method allows to solve the well known Curie-Weiss model. For this model a simple version of the method contains most of the crucial ingredients. The present rigorous method of solution is new and perhaps simpler

show abstract

“…In [30] several of the claims from [29] are examined and challenged. In [31] useful methods for the computation of information theoretic quantities are proposed for several deep neural network models.…”

Section: Related Workmentioning

confidence: 99%

Lautum Regularization for Semi-Supervised Transfer Learning

Jakubovitz

Rodrigues

Giryes

2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

Transfer learning is a very important tool in deep learning as it allows propagating information from one "source dataset" to another "target dataset", especially in the case of a small number of training examples in the latter. Yet, discrepancies between the underlying distributions of the source and target data are commonplace and are known to have a substantial impact on algorithm performance. In this work we suggest a novel information theoretic approach for the analysis of the performance of deep neural networks in the context of transfer learning. We focus on the task of semi-supervised transfer learning, in which unlabeled samples from the target dataset are available during the network training on the source dataset.Our theory suggests that one may improve the transferability of a deep neural network by imposing a Lautum information based regularization that relates the network weights to the target data. We demonstrate the effectiveness of the proposed approach in various transfer learning experiments.

show abstract

Entropy and mutual information in models of deep neural networks*

Cited by 102 publications

References 42 publications

Inference With Deep Generative Priors in High Dimensions

Inference With Deep Generative Priors in High Dimensions

The adaptive interpolation method for proving replica formulas. Applications to the Curie–Weiss and Wigner spike models

Lautum Regularization for Semi-Supervised Transfer Learning

Contact Info

Product

Resources

About