Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction

Abstract

Recently there has been significant theoretical progress on understanding the convergence and generalization of gradient-based methods on nonconvex losses with overparameterized models. Nevertheless, many aspects of optimization and generalization and in particular the critical role of small random initialization are not fully understood. In this paper, we take a step towards demystifying this role by proving that small random initialization followed by a few iterations of gradient descent behaves akin to popular spectral methods. We also show that this implicit spectral bias from small random initialization, which is provably more prominent for overparameterized models, also puts the gradient descent iterations on a particular trajectory towards solutions that are not only globally optimal but also generalize well. Concretely, we focus on the problem of reconstructing a low-rank matrix from a few measurements via a natural nonconvex formulation. In this setting, we show that the trajectory of the gradient descent iterations from small random initialization can be approximately decomposed into three phases: (I) a spectral or alignment phase where we show that that the iterates have an implicit spectral bias akin to spectral initialization allowing us to show that at the end of this phase the column space of the iterates and the underlying low-rank matrix are sufficiently aligned, (II) a saddle avoidance/refinement phase where we show that the trajectory of the gradient iterates moves away from certain degenerate saddle points, and (III) a local refinement phase where we show that after avoiding the saddles the iterates converge quickly to the underlying low-rank matrix. Underlying our analysis are insights for the analysis of overparameterized nonconvex optimization schemes that may have implications for computational problems beyond low-rank reconstruction.

Algorithmic Regularization in Model-free Overparametrized Asymmetric Matrix Factorization

“…Similar phenomena have been empirically observed in many other nonconvex problems, where vanilla gradient descent, when coupled with small random initialization (SRI) and early stopping (ES), has good generalization performance even with overpametrization due to the algorithmic regularization effect of SRI annd ES [Woodworth et al, 2020, Ghorbani et al, 2020, Prechelt, 1998, Wang et al, 2021, Li et al, 2018, Stöger and Soltanolkotabi, 2021. This motivates us to study the following question: What is the general behavior of the gradient descent dynamic (GD-M) coupled with SRI and ES?…”

“…• Model-free setting. Most of existing analysis considers the setting where X is (exactly or approximately) low-rank with a sufficiently large singular value gap δ [Li et al, 2018, Zhuo et al, 2021, Ye and Du, 2021, Fan et al, 2020, Stöger and Soltanolkotabi, 2021.…”

Global Convergence of Sub-gradient Method for Robust Matrix Recovery: Small Initialization, Noisy Measurements, and Over-parameterization

“…Based on our simulations, we observed that SubGM with small random initialization behaves almost the same as SubGM with spectral initialization. Therefore, we conjecture that small random initialization followed by a few iterations of SubGM is in fact equivalent to spectral initialization; a similar result has been recently proven by Stöger and Soltanolkotabi [31] for gradient descent on 2 -loss. We consider a rigorous verification of this conjecture as an enticing challenge for future research.…”

“…Therefore, the desirable performance of SubGM with small initialization can be attributed to its implicit regularization property. In particular, we show that small initialization of SubGM is akin to implicitly regularizing the redundant rank of the over-parameterized model, thereby avoiding overfitting; a recent work [31] has shown a similar property for the gradient descent algorithm on the noiseless matrix recovery with 2 -loss.…”

“…It has been recently shown that the trajectories picked up by gradient-based algorithms benefit from implicit regularization [17], or behave non-monotonically over short timescales, yet consistently improve over long timescales [9]. In the context of overparameterized low-rank matrix recovery with 2 -loss, Li et al [22] and Stöger and Soltanolkotabi [31] use trajectory analysis to show that GD with small initialization can recover the ground truth, provided that the measurements are noiseless. Zhuo et al [42] extend this result to the noisy setting, showing that GD converges to a minimax optimal solution at a sublinear rate, and with a number of samples that scale with the search rank.…”