Benign Underfitting of Stochastic Gradient Descent

Abstract

We study to what extent may stochastic gradient descent (SGD) be understood as a "conventional" learning rule that achieves generalization performance by obtaining a good fit to training data. We consider the fundamental stochastic convex optimization framework, where (one pass, without -replacement) SGD is classically known to minimize the population risk at rate O(1/ √ n), and prove that, surprisingly, there exist problem instances where the SGD solution exhibits both empirical risk and generalization gap of Ω(1). Consequently, it turns out that SGD is not algorithmically stable in any sense, and its generalization ability cannot be explained by uniform convergence or any other currently known generalization bound technique for that matter (other than that of its classical analysis). We then continue to analyze the closely related with-replacement SGD, for which we show that an analogous phenomenon does not occur and prove that its population risk does in fact converge at the optimal rate. Finally, we interpret our main results in the context of without-replacement SGD for finite-sum convex optimization problems, and derive upper and lower bounds for the multi-epoch regime that significantly improve upon previously known results.

Making Progress Based on False Discoveries

“…Together with Theorem 2.1 we obtain the following corollary We stress again, that, in contrast with these results, an optimization algorithm can correctly minimize the true loss using no more than Õ(1/ε 2 ) iterations and Õ(1/ε 2 ) examples [17].…”

“…It is also worth mentioning that recently Koren et al [17] showed that, in adaptive data analysis terminology, SGD is an example to a non post-hoc generalizing algorithm in the following sense: It can be shown that the output parameter w S provided by SGD may minimize the population loss, but there is a constant gap between the empirical and population loss at w S .…”