Robust Recovery via Implicit Bias of Discrepant Learning Rates for Double Over-parameterization

Abstract

Recent advances have shown that implicit bias of gradient descent on overparameterized models enables the recovery of low-rank matrices from linear measurements, even with no prior knowledge on the intrinsic rank. In contrast, for robust low-rank matrix recovery from grossly corrupted measurements, overparameterization leads to overfitting without prior knowledge on both the intrinsic rank and sparsity of corruption. This paper shows that with a double overparameterization for both the low-rank matrix and sparse corruption, gradient descent with discrepant learning rates provably recovers the underlying matrix even without prior knowledge on neither rank of the matrix nor sparsity of the corruption. We further extend our approach for the robust recovery of natural images by over-parameterizing images with deep convolutional networks. Experiments show that our method handles different test images and varying corruption levels with a single learning pipeline where the network width and termination conditions do not need to be adjusted on a case-by-case basis. Underlying the success is again the implicit bias with discrepant learning rates on different over-parameterized parameters, which may bear on broader applications.

Early Stopping for Deep Image Prior

“…For image denoising, we also systematically evaluate ES-WMV on major variants of DIP that try to mitigate overfitting, including DD [14] 1 , DIP-TV [4] 2 , GP-DIP [5] 3 , and demonstrate ES-WMV as a reliable helper to detect good ES points, whether these methods succeed or not in removing the overfitting. We also compare ES-WMV with major competing methods, including SB [34], DF-STE [20] 4 , SV-ES [23] 5 and DOP [45] 6 .…”

“…• Noise modeling: [45] models sparse additive noise as an explicit term in their optimization objective. [20] designs Gaussian-and Shot-specific regularizers and ES criteria.…”