Sharp Concentration Results for Heavy-Tailed Distributions

Bakhshizadeh, Milad; Maleki, Arian; Peña, Víctor H. de la

doi:10.48550/arxiv.2003.13819

Cited by 4 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We will bound the right tail of i X i and the left tail is bounded similarly. Using the same argument as Lemma 3(a) from [BMP20], one has lim L→∞ c L = Var(X). Further, it follows from definition that c L is bounded in any bounded subset of (0, ∞) and that c L is a continuous function of Notice that from the condition that I(t) ≥ t γ /C for some C > 0, it follows that…”

Section: Sums Of Heavy Tailed Random Variablesmentioning

confidence: 92%

See 1 more Smart Citation

A bounded-noise mechanism for differential privacy

Dagan¹,

Kur²

2020

Preprint

View full text Add to dashboard Cite

Answering multiple counting queries is one of the best-studied problems in differential privacy. Its goal is to output an approximation of the averagewhile preserving the privacy with respect to any x (i) . We present an (ǫ, δ)-private mechanism with optimal ℓ∞ error for most values of δ. This result settles the conjecture of Steinke and Ullman [2020] for the these values of δ. Our algorithm adds independent noise of bounded magnitude to each of the k coordinates, while prior solutions relied on unbounded noise such as the Laplace and Gaussian mechanisms.

show abstract

Section: Sums Of Heavy Tailed Random Variablesmentioning

confidence: 92%

“…Next, we would like to analyze concentration for sums of independent instances of f ′ DE (η i ) and f ′′ DE (η i ). For that purpose, we use a general result on concentration of sums of independent random variables, which follows from [BMP20]. (The proof appears in Section 4.3) Lemma 4.5.…”

Section: Proofmentioning

confidence: 99%

A bounded-noise mechanism for differential privacy

Dagan¹,

Kur²

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…The proof is given in Appendix C.2 and uses concentration results on heavy tailed distributions (Bakhshizadeh et al, 2020) to bound the operator norm of the matrix…”

Section: Effective Noise Levelmentioning

confidence: 99%

Trace norm regularization for multi-task learning with scarce data

Boursier¹,

Konobeev²,

Flammarion³

2022

Preprint

View full text Add to dashboard Cite

Multi-task learning leverages structural similarities between multiple tasks to learn despite very few samples. Motivated by the recent success of neural networks applied to data-scarce tasks, we consider a linear low-dimensional shared representation model. Despite an extensive literature, existing theoretical results either guarantee weak estimation rates or require a large number of samples per task. This work provides the first estimation error bound for the trace norm regularized estimator when the number of samples per task is small. The advantages of trace norm regularization for learning data-scarce tasks extend to meta-learning and are confirmed empirically on synthetic datasets.

show abstract

“…Heavy-tailed distributions are distributions whose moment-generating function diverges for all positive parameterizations ( [14]), and have been used as models of correlated systems ( [15,16,17,18,19,20]). In this work, we assume that the fully trained weight matrices of a fully connected neural network (FCNN) have power-law structure ( [11] and [1]), and using the compression framework of [2], prove bounds on generalization given this structure.…”

Section: Introductionmentioning

confidence: 99%

“…This results in a nonvacuous generalization bound, i.e., a bound that is less than one. The key idea of how to compress these matrices was inspired by the work of [16] and [15]. The work will be structured as follows:…”

Section: Introductionmentioning

confidence: 99%

Compressing Heavy-Tailed Weight Matrices for Non-Vacuous Generalization Bounds

Shin

2021

Preprint

View full text Add to dashboard Cite

Heavy-tailed distributions have been studied in statistics, random matrix theory, physics, and econometrics as models of correlated systems, among other domains. Further, heavy-tail distributed eigenvalues of the covariance matrix of the weight matrices in neural networks have been shown to empirically correlate with test set accuracy in several works (e.g. [1]), but a formal relationship between heavy-tail distributed parameters and generalization bounds was yet to be demonstrated. In this work, the compression framework of [2] is utilized to show that matrices with heavy-tail distributed matrix elements can be compressed, resulting in networks with sparse weight matrices. Since the parameter count has been reduced to a sum of the non-zero elements of sparse matrices, the compression framework allows us to bound the generalization gap of the resulting compressed network with a non-vacuous generalization bound. Further, the action of these matrices on a vector is discussed, and how they may relate to compression and resilient classification is analyzed. * Please see the acknowledgements section for affiliation details.Preprint. Under review.

show abstract

Sharp Concentration Results for Heavy-Tailed Distributions

Cited by 4 publications

References 0 publications

A bounded-noise mechanism for differential privacy

A bounded-noise mechanism for differential privacy

Trace norm regularization for multi-task learning with scarce data

Compressing Heavy-Tailed Weight Matrices for Non-Vacuous Generalization Bounds

Contact Info

Product

Resources

About