Implicit Deep Learning

Ghaoui, Laurent El; Gu, Fangda; Travacca, Bertrand; Askari, Armin; Tsai, Alicia Y.

doi:10.1137/20m1358517

Cited by 55 publications

(52 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, for the training procedure, this condition is relaxed to the convex condition A ∞ < 1. It is easy to see that our well-posedness condition in Corollary 5(i) is less conservative than the condition λ pf (|A|) < 1 and its convex relaxation of the form A ∞ < 1 proposed in [El Ghaoui et al, 2019].…”

Section: Contraction Analysis Of Implicit Neural Networkmentioning

confidence: 85%

“…In [El Ghaoui et al, 2019] a well-posedness condition of the form λ pf (|A|) < 1 is proposed, where |A| denotes the entrywise absolute value of the matrix A and λ pf denotes the Perron-Frobenius eigenvalue. However, for the training procedure, this condition is relaxed to the convex condition A ∞ < 1.…”

Section: Contraction Analysis Of Implicit Neural Networkmentioning

confidence: 99%

“…Backpropagation of gradients via average iteration From [El Ghaoui et al, 2019] we now show how the average iteration can be used to perform backpropagation via the implicit function theorem. For simplicity, we assume that each activation function φ i is differentiable and consider mini-batches of size 1, i.e., we have…”

Section: Training Implicit Neural Networkmentioning

confidence: 99%

“…Implicit neural networks are infinite-depth learning models with layers defined implicitly through a fixed-point equation. Examples of implicit neural networks include deep equilibrium models [Bai et al, 2019] and implicit deep learning models [El Ghaoui et al, 2019]. Implicit networks can be considered as generalizations of feedforward neural networks with input-injected weight tying, i.e., training parameters are transferable between layers.…”

Section: Introductionmentioning

confidence: 99%

“…Indeed, without additional assumptions, their fixed-point equation may not have a unique solution and the numerical algorithms for finding their solutions might not converge. Several recent works in the literature have focused on studying well-posedness and convergence of the fixed-point equations of implicit networks using frameworks such as monotone operator theory [Winston and Kolter, 2020], contraction theory [El Ghaoui et al, 2019], and a mixture of both [Revay et al, 2020]. Despite several insightful results, important questions about conditions for well-posedness of implicit networks and efficient algorithms that converge to their solutions are still open.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Robust Implicit Networks via Non-Euclidean Contractions

Jafarpour¹,

Davydov²,

Proskurnikov³

et al. 2021

Preprint

View full text Add to dashboard Cite

Implicit neural networks, a.k.a., deep equilibrium networks, are a class of implicit-depth learning models where function evaluation is performed by solving a fixed point equation. They generalize classic feedforward models and are equivalent to infinite-depth weight-tied feedforward networks. While implicit models show improved accuracy and significant reduction in memory consumption, they can suffer from ill-posedness and convergence instability.This paper provides a new framework to design well-posed and robust implicit neural networks based upon contraction theory for the non-Euclidean norm ∞. Our framework includes (i) a novel condition for wellposedness based on one-sided Lipschitz constants, (ii) an average iteration for computing fixed-points, and (iii) explicit estimates on input-output Lipschitz constants. Additionally, we design a training problem with the well-posedness condition and the average iteration as constraints and, to achieve robust models, with the inputoutput Lipschitz constant as a regularizer. Our ∞ well-posedness condition leads to a larger polytopic training search space than existing conditions and our average iteration enjoys accelerated convergence. Finally, we perform several numerical experiments for function estimation and digit classification through the MNIST data set. Our numerical results demonstrate improved accuracy and robustness of the implicit models with smaller input-output Lipschitz bounds.

show abstract

Section: Contraction Analysis Of Implicit Neural Networkmentioning

confidence: 85%

Section: Contraction Analysis Of Implicit Neural Networkmentioning

confidence: 99%

Section: Training Implicit Neural Networkmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations