Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD

Nikolakakis, Konstantinos E.; Haddadpour, Farzin; Karbasi, Amin; Kalogerias, Dionysios S.

doi:10.48550/arxiv.2204.12446

Cited by 3 publications

(13 citation statements)

References 18 publications

(33 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…( 5) under regime T = O (n), this is suboptimal in the rate of n, but holds for a more general choice of T . Similar results hold for GD under realizable smooth SCO (Nikolakakis et al, 2022;Schliserman and Koren, 2022) and are summarized in Table 1.…”

Section: Realizable Smooth Scosupporting

confidence: 68%

“…It is widely employed in stochastic optimization literature to improve the convergence rate of SGD and GD under overparameterized or realizable setting (Vaswani et al, 2019). Recent papers (Lei and Ying, 2020;Schliserman and Koren, 2022;Nikolakakis et al, 2022) focused on the generalization bound under realizable smooth SCO also suggest that such an assumption improves the sample complexity upper bounds beyond the rate of O (1/ √ n), which is the well-known result for GD and SGD without the realizable assumption (Hardt et al, 2016).…”

Section: Realizable Smooth Scomentioning

confidence: 86%

“…However, less is known when the function is differentiable and smooth. Indeed, while upper bounds have been well-established in literature (Hardt et al, 2016;Lei and Ying, 2020;Nikolakakis et al, 2022), the optimality of these results need to be certified by corresponding lower bound constructions. In this work, we focus on the smooth SCO setting and make the following assumptions.…”

Section: Smooth Stochastic Convex Optimizationmentioning

confidence: 99%

“…For each setting, we provide lower bounds for the excess risk of GD, SGD, and corresponding best possible sample complexity bounds. † The bound in η, T and n is not explicitly stated in Nikolakakis et al (2022). For the expression, please refer to a derivation in Appendix D.3.…”

Section: Realizable Smooth Scomentioning

confidence: 99%

“…Our aim is to understand whether longer training inevitably leads to overfitting in smooth SCO. Several recent works have shown fast convergence rates in test error when the number of iterations is not too large under the realizable condition (Lei and Ying, 2020;Nikolakakis et al, 2022;Schliserman and Koren, 2022). Our work provides the first tight lower bounds in these scenarios.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Lower Generalization Bounds for GD and SGD in Smooth Stochastic Convex Optimization

Zhang¹,

Teng²,

Zhang³

2023

Preprint

View full text Add to dashboard Cite

Recent progress was made in characterizing the generalization error of gradient methods for general convex loss by the learning theory community. In this work, we focus on how training longer might affect generalization in smooth stochastic convex optimization (SCO) problems. We first provide tight lower bounds for general non-realizable SCO problems. Furthermore, existing upper bound results suggest that sample complexity can be improved by assuming the loss is realizable, i.e. an optimal solution simultaneously minimizes all the data points. However, this improvement is compromised when training time is long and lower bounds are lacking. Our paper examines this observation by providing excess risk lower bounds for gradient descent (GD) and stochastic gradient descent (SGD) in two realizable settings: 1) realizable with T = O (n), and (2) realizable with T = Ω (n), where T denotes the number of training iterations and n is the size of the training dataset. These bounds are novel and informative in characterizing the relationship between T and n. In the first small training horizon case, our lower bounds almost tightly match and provide the first optimal certificates for the corresponding upper bounds. However, for the realizable case with T = Ω (n), a gap exists between the lower and upper bounds. We provide a conjecture to address this problem, that the gap can be closed by improving upper bounds, which is supported by our analyses in one-dimensional and linear regression scenarios.

show abstract

Section: Realizable Smooth Scosupporting

confidence: 68%

Section: Realizable Smooth Scomentioning

confidence: 86%

Section: Smooth Stochastic Convex Optimizationmentioning

confidence: 99%

Section: Realizable Smooth Scomentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Lower Generalization Bounds for GD and SGD in Smooth Stochastic Convex Optimization

Zhang¹,

Teng²,

Zhang³

2023

Preprint

View full text Add to dashboard Cite

show abstract

Towards Stability and Generalization Bounds in Decentralized Minibatch Stochastic Gradient Descent

Wang,

Chen

2024

AAAI

View full text Add to dashboard Cite

Decentralized Stochastic Gradient Descent (D-SGD) represents an efficient communication approach tailored for mastering insights from vast, distributed datasets. Inspired by parallel optimization paradigms, the incorporation of minibatch serves to diminish variance, consequently expediting the optimization process. Nevertheless, as per our current understanding, the existing literature has not thoroughly explored the learning theory foundation of Decentralized Minibatch Stochastic Gradient Descent (DM-SGD). In this paper, we try to address this theoretical gap by investigating the generalization properties of DM-SGD. We establish the sharper generalization bounds for the DM-SGD algorithm with replacement (without replacement) on (non)convex and (non)smooth cases. Moreover, our results consistently recover to the results of Centralized Stochastic Gradient Descent (C-SGD). In addition, we derive generalization analysis for Zero-Order (ZO) version of DM-SGD.

show abstract

Improving Variational Methods for Representation Learning and Generative Modelling of Speech

Braithwaite¹

View full text Add to dashboard Cite

Hearing aids are clinically proven to improve the quality of life of hearing-impaired individuals. However, despite this, many people who could benefit from a hearing aid do not use one. The most common reason for this is that they perform poorly in noisy environments. Machine learning presents a collection of powerful methods for attenuating noise in speech signals. Moreover, state-of-the-art hearing aids are being designed with more powerful processors that can run machine learning algorithms. Several key factors make the deployment of machine learning to hearing aids challenging. Firstly, many state-of-the-art machine learning methods have high resource requirements (e.g., GPUs or large amounts of memory). Hence, these resource-intensive models cannot be run in real-time in low-complexity systems. Secondly, methods need to be causal (i.e., do not depend on access to future information). More generally, an issue is that the objectives that speech enhancement/separation models typically optimise (e.g., L1 or L2) correlate poorly with perceptual quality. The goal of this thesis is to develop improved models for attenuating noise in speech signals. Specifically, we aim to develop more principled models and better understand existing frameworks. Overall, the objective is to develop low-complexity models for attenuating noise in speech signals. We begin by developing an improved autoencoder-based generative model, which we name the Bounded Information Rate Variational Autoencoder. This model resolves the posterior collapse issue with Variational Autoencoders by pre-specifying the information rate between the data and latent variables. In a further extension of this model, we relax the constraints on the distribution of latent variables. Hence, the variance of this distribution is constrained, but not the shape. We name this model the Variance Constrained Autoencoder (VCAE). The VCAE outperforms other state-of-the-art generative models in both reconstruction and generation quality. In subsequent work, we applied the Variance Constrained Autoencoder to speech enhancement. The model is trained to reconstruct a clean target signal from the noisy input. Importantly, rather than enhancing large segments, we enhanced short blocks ($\tilde 4$ ms). This allowed our system to have a lower computation complexity and higher subjective performance than other state-of-the-art approaches. Our final contribution is an empirical study of a speech separation system called Conv-TasNet. This model separates speech signals by applying multiplicative masks to the mixture signal in a learned over-parameterised transform domain. The phenomenon of interest is the structure of the learned transform. Specifically, an over-parameterised frequency domain transform is learned, where more emphasis is placed on the regions with more signal energy. Hence, the transformation resembles something that is perceptually motivated. In the empirical study of Conv-TasNet, we first provide empirical evidence that shows the transform learned by Conv-TasNet prefers uncorrelated coefficients. This explains why a frequency domain transform is learned. Moreover, we show that performance can be improved over the base system by reducing the correlation of the transform space coefficients. In the second part of this contribution, we demonstrate a link between the flatness of the filterbank and the overall separation performance. Furthermore, we show that the density of the vectors affects the flatness of the frame. In other words, we provide evidence that the perceptual structure of the learned filterbank results from an implicit bias in stochastic gradient descent training.

show abstract

Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for Full-Batch GD

Cited by 3 publications

References 18 publications

Lower Generalization Bounds for GD and SGD in Smooth Stochastic Convex Optimization

Lower Generalization Bounds for GD and SGD in Smooth Stochastic Convex Optimization

Towards Stability and Generalization Bounds in Decentralized Minibatch Stochastic Gradient Descent

Improving Variational Methods for Representation Learning and Generative Modelling of Speech

Contact Info

Product

Resources

About