Information-Theoretic Generalization Bounds for Stochastic Gradient Descent

Neu, Gergely; Dziugaite, Gintare Karolina; Haghifam, Mahdi; Roy, Daniel M.

doi:10.48550/arxiv.2102.00931

Cited by 4 publications

(29 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The bound was subsquently improved by Negrea et al (2019); Haghifam et al (2020); Rodríguez-Gálvez et al (2020); Wang et al (2021). Inspired by the work of Pensia et al (2018), Neu (2021) presents an information-theoretic analysis of the models trained with SGD. The analysis of Neu (2021) constructs an auxiliary weight process parallel to SGD training and upper-bounds the generalization error through this auxiliary process.…”

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…Our development builds upon that of Neu (2021). Following the same construction of the auxiliary weight process in Neu (2021), we present upper bounds of generalization error that improve upon Neu (2021) in two ways. This improvement is mainly due to removing an unnecessary term in the bound of Neu (2021), which the author refers to as "local gradient sensitivity", by invoking the HWI inequality (Raginsky & Sason, 2018) (i.e., Lemma 4 of this paper).…”

Section: Introductionmentioning

confidence: 99%

“…Following the same construction of the auxiliary weight process in Neu (2021), we present upper bounds of generalization error that improve upon Neu (2021) in two ways. This improvement is mainly due to removing an unnecessary term in the bound of Neu (2021), which the author refers to as "local gradient sensitivity", by invoking the HWI inequality (Raginsky & Sason, 2018) (i.e., Lemma 4 of this paper). Additionally, the improvement also benefits from replacing a samplelevel mutual information term in Neu (2021) with an instance-level mutual information term, exploiting a recent result of Bu et al (2020).…”

Section: Introductionmentioning

confidence: 99%

“…

…”

mentioning

confidence: 99%

See 4 more Smart Citations

On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications

Wang¹,

Mao²

2021

Preprint

View full text Add to dashboard Cite

This paper follows up on a recent work of Neu (2021) and presents new and tighter informationtheoretic upper bounds for the generalization error of machine learning models, such as neural networks, trained with SGD. We apply these bounds to analyzing the generalization behaviour of linear and twolayer ReLU networks. Experimental study based on these bounds provide some insights on the SGD training of neural networks. They also point to a new and simple regularization scheme which we show performs comparably to the current state of the art.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…

…”

mentioning

confidence: 99%

See 3 more Smart Citations

On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications

Wang¹,

Mao²

2021

Preprint

View full text Add to dashboard Cite

show abstract

Information-theoretic generalization bounds for black-box learning algorithms

Harutyunyan,

Raginsky,

Steeg

et al. 2021

Preprint

View full text Add to dashboard Cite

We derive information-theoretic generalization bounds for supervised learning algorithms based on the information contained in predictions rather than in the output of the training algorithm. These bounds improve over the existing informationtheoretic bounds, are applicable to a wider range of algorithms, and solve two key challenges: (a) they give meaningful results for deterministic algorithms and (b) they are significantly easier to estimate. We show experimentally that the proposed bounds closely follow the generalization gap in practical scenarios for deep learning.

show abstract

User-friendly introduction to PAC-Bayes bounds

Alquier¹

2021

Preprint

View full text Add to dashboard Cite

Aggregated predictors are obtained by making a set of basic predictors vote according to some weights, that is, to some probability distribution.Randomized predictors are obtained by sampling in a set of basic predictors, according to some prescribed probability distribution.Thus, aggregated and randomized predictors have in common that they are not defined by a minimization problem, but by a probability distribution on the set of predictors. In statistical learning theory, there is a set of tools designed to understand the generalization ability of such procedures: PAC-Bayesian or PAC-Bayes bounds.Since the original PAC-Bayes bounds [163,124], these tools have been considerably improved in many directions (we will for example describe a simplified version of the localization technique of [39,41] that was missed by the community, and later rediscovered as "mutual information bounds"). Very recently, PAC-Bayes bounds received a considerable attention: for example there was workshop on PAC-Bayes at NIPS 2017, (Almost) 50 Shades of Bayesian Learning: PAC-Bayesian trends and insights, organized by B. Guedj, F. Bach and P. Germain. One of the reasons of this recent success is the successful application of these bounds to neural networks [65].An elementary introduction to PAC-Bayes theory is still missing. This is an attempt to provide such an introduction. This is a preliminary version. If you find any typo/mistake, if you think your work is not cited and should be, please let me know, and I will update the tutorial accordingly. Since 1st version: fixed (minor) problems in Theorem 4.5, in Lemma 4.6 and in Subsection 6.5.2, fixed many typos (including some in the proof of Theorem 4.3), included ref. [26,131,114].1 I don't want to scare the reader with measurability conditions, as I will not check them in this tutorial anyway. Here, the exact condition to ensure that what follows is well defined is that for any A ∈ T , the function ((x 1 , y 1 ), . . . , (x n , y n )) → [ρ((x 1 , y 1 ), . . . , (x n , y n ))] (A) is measurable. That is, ρ is a regular conditional probability.2 See the title of van Erven's tutorial [175]: "PAC-Bayes mini-tutorial: a continuous union bound". Note, however, that it is argued by Catoni in [41] that PAC-Bayes bounds are actually more than that, we will come back to this in Section 4.

show abstract

Information-Theoretic Generalization Bounds for Stochastic Gradient Descent

Cited by 4 publications

References 0 publications

On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications

On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications

Information-theoretic generalization bounds for black-box learning algorithms

User-friendly introduction to PAC-Bayes bounds

Contact Info

Product

Resources

About