Information-Theoretic Bounds on the Moments of the Generalization Error of Learning Algorithms

Abstract

Generalization error bounds are critical to understanding the performance of machine learning models. In this work, building upon a new bound of the expected value of an arbitrary function of the population and empirical risk of a learning algorithm, we offer a more refined analysis of the generalization behaviour of a machine learning models based on a characterization of (bounds) to their generalization error moments. We discuss how the proposed bounds -which also encompass new bounds to the expected generalization errorrelate to existing bounds in the literature. We also discuss how the proposed generalization error moment bounds can be used to construct new generalization error high-probability bounds.

Characterizing and Understanding the Generalization Error of Transfer Learning with Gibbs Algorithm

“…[28] provides tighter bounds by considering the individual sample mutual information, [25,29] propose using chaining mutual information, and [30,31,32] advocate the conditioning and processing techniques. Information-theoretic generalization error bounds using other information quantities are also studied, such as, f -divergence [33], α-Rényi divergence and maximal leakage [34,35], and Jensen-Shannon divergence [36,37]. Using rate-distortion theory, [38,39,40] provide information-theoretic generalization error upper bounds for model misspecification and model compression.…”