Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias Towards Low Rank

Chou, Hung-Hsu; Gieshoff, Carsten; Maly, Johannes; Rauhut, Holger

doi:10.2139/ssrn.4139166

Cited by 3 publications

(4 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In fact, the overparametrized gradient descent provides a tuning free alternative to established algorithms like LASSO and Basis Pursuit. In a similar manner our present contribution stems from the insight that in most of the above papers [1,2,17,18] the signs of components do not change over time when gradient flow is applied. Instead of viewing this feature as an obstacle, cf.…”

Section: Related Work -Overparameterization and Implicit Regularizationmentioning

confidence: 61%

“…Because ∂ t D F (z + , x(t)) is identical for all z + ∈ S + (the second line of (17) does not depend on the choice of z + ), the difference…”

Section: Discussionmentioning

confidence: 99%

“…However, due to the complexity of the model class, there is still no corresponding thorough theoretical analysis/understanding available. To close the gap between theory and practice two simplified "training" models have been proposed and analyzed in a great number of works: vector factorization [38,75,68,71,45,18] and matrix factorization [1,2,17,29,30,32,35,36,52,53,57,61,64,70,72]. Essentially, all of these works agree in the point that, if initialized close to zero and applied to a plain least-squares formulation of factorized shape, cf.…”

Section: Related Work -Overparameterization and Implicit Regularizationmentioning

confidence: 99%

See 2 more Smart Citations

Non-negative Least Squares via Overparametrization

Chou¹,

Maly²,

Verdun³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

In many applications, solutions of numerical problems are required to be non-negative, e.g., when retrieving pixel intensity values or physical densities of a substance. In this context, nonnegative least squares (NNLS) is a ubiquitous tool, especially when seeking sparse solutions of high-dimensional statistical problems. Despite vast efforts since the seminal work of Lawson and Hanson in the '70s, the non-negativity assumption is still an obstacle for the theoretical analysis and scalability of many off-the-shelf solvers. In the different context of deep neural networks, we recently started to see that the training of overparametrized models via gradient descent leads to surprising generalization properties and the retrieval of regularized solutions. In this paper, we prove that, by using an overparametrized formulation, NNLS solutions can reliably be approximated via vanilla gradient flow. We furthermore establish stability of the method against negative perturbations of the ground-truth. Our simulations confirm that this allows to use vanilla gradient descent as a novel and scalable numerical solver for NNLS.

show abstract

Section: Related Work -Overparameterization and Implicit Regularizationmentioning

confidence: 61%

“…Because ∂ t D F (z + , x(t)) is identical for all z + ∈ S + (the second line of (17) does not depend on the choice of z + ), the difference…”

Section: Discussionmentioning

confidence: 99%

Section: Related Work -Overparameterization and Implicit Regularizationmentioning

confidence: 99%

See 1 more Smart Citation

Non-negative Least Squares via Overparametrization

Chou¹,

Maly²,

Verdun³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Also, the results show that the neural networks were robust against noise even when the neural networks had many parameters. The neural networks' robustness against noise is probably due to implicit bias, in which over‐parametrized neural networks tend to learn a simple structure from data (Chou et al., 2024). This property is not trivial because traditional approaches (e.g., using linear interpolation functions to represent the constitutive relations) would be overfitted to the noisy data and require some regularization techniques.…”

Section: Inverse Modelingmentioning

confidence: 99%

Learning Constitutive Relations From Soil Moisture Data via Physically Constrained Neural Networks

Bandai,

Ghezzehei,

Jiang

et al. 2024

Water Resources Research

View full text Add to dashboard Cite

The constitutive relations of the Richardson‐Richards equation encode the macroscopic properties of soil water retention and conductivity. These soil hydraulic functions are commonly represented by models with a handful of parameters. The limited degrees of freedom of such soil hydraulic models constrain our ability to extract soil hydraulic properties from soil moisture data via inverse modeling. We present a new free‐form approach to learning the constitutive relations using physically constrained neural networks. We implemented the inverse modeling framework in a differentiable modeling framework, JAX, to ensure scalability and extensibility. For efficient gradient computations, we implemented implicit differentiation through a nonlinear solver for the Richardson‐Richards equation. We tested the framework against synthetic noisy data and demonstrated its robustness against varying magnitudes of noise and degrees of freedom of the neural networks. We applied the framework to soil moisture data from an upward infiltration experiment and demonstrated that the neural network‐based approach was better fitted to the experimental data than a parametric model and that the framework can learn the constitutive relations.

show abstract