Information-Theoretic Bounds on the Moments of the Generalization Error of Learning Algorithms

Aminian, Gholamali; Toni, Laura; Rodrigues, Miguel R. D.

doi:10.1109/isit45174.2021.9518043

Cited by 8 publications

(5 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…PAC-Bayesian generalization error bounds: First proposed by [6], [51], [52], PAC-Bayesian analysis provides high probability bounds on the generalization error in terms of KL divergence between the data-dependent posterior induced by the learning algorithm and a data-free prior that can be chosen arbitrarily [53]. There are multiple ways to generalize the standard PAC-Bayesian bounds, including using different information measures other than the KL divergence [54]- [58] and considering data-dependent priors (prior depends on the training data) [13], [59]- [63] or distribution-dependent priors (prior depends on data-generating distribution) [64]- [67]. In [68], a more general PAC-Bayesian framework is proposed, which provides a high probability bound on the convex function of the expected population and empirical risk with respect to the posterior distribution, whereas in [69] the connection between Bayesian inference and PAC-Bayesian theorem is explored by considering Gibbs posterior and negative log loss function.…”

Section: F Other Related Workmentioning

confidence: 99%

Information-Theoretic Characterizations of Generalization Error for the Gibbs Algorithm

Aminian,

Bu,

Toni

et al. 2024

IEEE Trans. Inform. Theory

Self Cite

View full text Add to dashboard Cite

Various approaches have been developed to upper bound the generalization error of a supervised learning algorithm. However, existing bounds are often loose and even vacuous when evaluated in practice. As a result, they may fail to characterize the exact generalization ability of a learning algorithm. Our main contributions are exact characterizations of the expected generalization error of the well-known Gibbs algorithm (a.k.a. Gibbs posterior) using different information measures, in particular, the symmetrized KL information between the input training samples and the output hypothesis. Our result can be applied to tighten existing expected generalization errors and PAC-Bayesian bounds. Our information-theoretic approach is versatile, as it also characterizes the generalization error of the Gibbs algorithm with a data-dependent regularizer and that of the Gibbs algorithm in the asymptotic regime, where it converges to the standard empirical risk minimization algorithm. Of particular relevance, our results highlight the role the symmetrized KL information plays in controlling the generalization error of the Gibbs algorithm.

show abstract

Section: F Other Related Workmentioning

confidence: 99%

Information-Theoretic Characterizations of Generalization Error for the Gibbs Algorithm

Aminian,

Bu,

Toni

et al. 2024

IEEE Trans. Inform. Theory

Self Cite

View full text Add to dashboard Cite

show abstract

“…Robust PLS: Robustness of PLS is a widely discussed issue in the self-training literature. [Aminian et al, 2022] propose information-theoretic PLS robust towards covariate shift. [Lienen and Hüllermeier, 2021] label instances in the form of sets of probability distributions (credal sets), weakening the reliance on a single distribution.…”

Section: Related Workmentioning

confidence: 99%

Approximate Bayes Optimal Pseudo-Label Selection

Rodemann¹,

Goschenhofer²,

Dorigatti³

et al. 2023

Preprint

View full text Add to dashboard Cite

Semi-supervised learning by self-training heavily relies on pseudo-label selection (PLS). The selection often depends on the initial model fit on labeled data. Early overfitting might thus be propagated to the final model by selecting instances with overconfident but erroneous predictions, often referred to as confirmation bias. This paper introduces BPLS, a Bayesian framework for PLS that aims to mitigate this issue. At its core lies a criterion for selecting instances to label: an analytical approximation of the posterior predictive of pseudo-samples. We derive this selection criterion by proving Bayes optimality of the posterior predictive of pseudo-samples. We further overcome computational hurdles by approximating the criterion analytically. Its relation to the marginal likelihood allows us to come up with an approximation based on Laplace's method and the Gaussian integral. We empirically assess BPLS for parametric generalized linear and non-parametric generalized additive models on simulated and realworld data. When faced with high-dimensional data prone to overfitting, BPLS outperforms traditional PLS methods. a a Code available at: https://anonymous.

show abstract

“…Information-theoretic generalization error bounds using other information quantities are also studied, such as α-Rényi divergence and maximal leakage (Esposito et al, 2021), Jensen-Shannon divergence (Aminian et al, 2021b), power divergence (Aminian et al, 2021c), and Wasserstein distance (Lopez and Jog, 2018;Wang et al, 2019). An exact characterization of the generalization error for the Gibbs algorithm is provided in (Aminian et al, 2021a).…”

Section: Related Workmentioning

confidence: 99%

An Information-theoretical Approach to Semi-supervised Learning under Covariate-shift

Aminian¹,

Abroshan²,

Khalili³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

A common assumption in semi-supervised learning is that the labeled, unlabeled, and test data are drawn from the same distribution. However, this assumption is not satisfied in many applications. In many scenarios, the data is collected sequentially (e.g., healthcare) and the distribution of the data may change over time often exhibiting socalled covariate shifts. In this paper, we propose an approach for semi-supervised learning algorithms that is capable of addressing this issue. Our framework also recovers some popular methods, including entropy minimization and pseudo-labeling. We provide new information-theoretical based generalization error upper bounds inspired by our novel framework. Our bounds are applicable to both general semi-supervised learning and the covariate-shift scenario. Finally, we show numerically that our method outperforms previous approaches proposed for semisupervised learning under the covariate shift.

show abstract

Information-Theoretic Bounds on the Moments of the Generalization Error of Learning Algorithms

Cited by 8 publications

References 12 publications

Information-Theoretic Characterizations of Generalization Error for the Gibbs Algorithm

Information-Theoretic Characterizations of Generalization Error for the Gibbs Algorithm

Approximate Bayes Optimal Pseudo-Label Selection

An Information-theoretical Approach to Semi-supervised Learning under Covariate-shift

Contact Info

Product

Resources

About