On the regularizing property of stochastic gradient descent

Abstract

Stochastic gradient descent (SGD) and its variants are among the most successful approaches for solving large-scale optimization problems. At each iteration, SGD employs an unbiased estimator of the full gradient computed from one single randomly selected data point. Hence, it scales well with problem size and is very attractive for handling truly massive dataset, and holds significant potentials for solving large-scale inverse problems. In this work, we rigorously establish its regularizing property under a priori early stopping rule for linear inverse problems, and also prove convergence rates under the canonical sourcewise condition. This is achieved by combining tools from classical regularization theory and stochastic analysis. Further, we analyze its preasymptotic weak and strong convergence behavior, in order to explain the fast initial convergence typically observed in practice. The theoretical findings shed insights into the performance of the algorithm, and are complemented with illustrative numerical experiments.

Latent Factor Modeling of Perceived Quality for Stereoscopic 3D Video Recommendation

“…The stochastic gradient descent algorithm [23,36,37] is used to optimize equation (3). For better accuracy in prediction, the algorithm loops through all ratings in the training data and estimates the model parameters.…”

Regularising linear inverse problems under unknown non-Gaussian noise

“…However, they do not give a rate of convergence, which remains an open problem. Numerically, we observe that the convergence rate obtained by the discrepancy principle is nearly order-optimal for low-regularity solutions, as the a priori rule in the regime in [JL19], and the performance is competitive with the standard Landweber method. Thus, the method is especially attractive for finding a low-accuracy solution.…”

“…Remark 3.2.3. The condition r < 1 is related to an apparent saturation phenomenon with SGD: for any ν > 1, the SGD iterate x δ k with a priori stopping can only achieve a convergence rate comparable with that for ν = 1 in the setting of Assumption 3.1.1, at least for the current analysis [JL19]. It remains unclear whether this is an intrinsic drawback of SGD or due to limitations of the proof technique.…”

“…Remark 3.1.3. The condition r < 1 is related to an apparent saturation phenomenon with SGD: for any ν > 1, the SGD iterate x δ k with a priori stopping can only achieve a convergence rate comparable with that for ν = 1 in the setting of Assumption 3.1.1, at least for the analysis in [JL19]. However, in the very recent preprint [JZZ20b] a refined convergence analysis is presented, showing that this saturation actually does not occur, if the initial step size c 0 is sufficiently small.…”

“…However, the mathematical theory in the lens of classical regularisation theory is rather incomplete, as it does not fit in the framework of filter-based regularisation. In the work [JL19] the regularising property of stochastic gradient descent was explored for the first time. Further, a convergence rate in the mean squared norm was derived, under suitable source type condition on the true solutionx.…”