Gradient Dynamics of Shallow Univariate ReLU Networks

Williams, Francis; Trager, Matthew; Silva, Claudio; Panozzo, Daniele; Zorin, Denis; Bruna, Joan

doi:10.48550/arxiv.1906.07842

Cited by 9 publications

(9 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Yet, in practical-sized NNs the spectrum of G t is neither constant nor it is similar to its initialization. Recent several studies explored its adaptive dynamics [4,19,20], yet most of the work was done for single or two layer NNs. Likewise, in [5,9] mathematical expressions for NTK dynamics were developed for a general NN architecture.…”

Section: Related Workmentioning

confidence: 99%

Neural Spectrum Alignment: Empirical Study

Kopitkov¹,

Indelman²

2019

Preprint

View full text Add to dashboard Cite

Expressiveness and generalization of deep models was recently addressed via the connection between neural networks (NNs) and kernel learning, where first-order dynamics of NN during a gradient-descent (GD) optimization were related to gradient similarity kernel, also known as Neural Tangent Kernel (NTK) [10]. In the majority of works this kernel is considered to be time-invariant [10,14], with its properties being defined entirely by NN architecture and independent of the learning task at hand. In contrast, in this paper we empirically explore these properties along the optimization and show that in practical applications the NN kernel changes in a very dramatic and meaningful way, with its top eigenfunctions aligning toward the target function learned by NN. Moreover, these top eigenfunctions serve sort of basis functions for NN output -a function represented by NN is spanned almost completely by them for the entire optimization process. Further, since the learning along top eigenfunctions is typically fast, their alignment with the target function improves the overall optimization performance. In addition, we study how the neural spectrum is affected by learning rate decay, typically done by practitioners, showing various trends in the kernel behavior. We argue that the presented phenomena may lead to a more complete theoretical understanding behind NN learning.

show abstract

Section: Related Workmentioning

confidence: 99%

Neural Spectrum Alignment: Empirical Study

Kopitkov¹,

Indelman²

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…It is interesting that even when allowing for more input arguments, the resultant learned nonlinearities still favor low-order quadratic functions (Figure 8b-d). This could be explained by an implicit bias toward smooth functions (Williams et al, 2019;Sahs et al, 2020) while still bending the input space to provide useful computations. Perhaps the learned nonlinearities are as random as possible while fulfilling these minimal conditions.…”

Section: Discussionmentioning

confidence: 99%

Two-argument activation functions learn soft XOR operations like cortical neurons

Yoon¹,

Orhan²,

Kim³

et al. 2021

Preprint

View full text Add to dashboard Cite

Neurons in the brain are complex machines with distinct functional compartments that interact nonlinearly. In contrast, neurons in artificial neural networks abstract away this complexity, typically down to a scalar activation function of a weighted sum of inputs.Here we emulate more biologically realistic neurons by learning canonical activation functions with two input arguments, analogous to basal and apical dendrites. We use a networkin-network architecture where each neuron is modeled as a multilayer perceptron with two inputs and a single output. This inner perceptron is shared by all units in the outer network. Remarkably, the resultant nonlinearities often produce soft XOR functions, consistent with recent experimental observations about interactions between inputs in human cortical neurons. When hyperparameters are optimized, networks with these nonlinearities learn faster and perform better than conventional ReLU nonlinearities with matched parameter counts, and they are more robust to natural and adversarial perturbations.

show abstract

“…For Neural Splines, the kernel norm favors smooth functions: It is proportional to curvature ( f K ≈ f ) for 1D curves [65] and to the Radon transform of the Laplacian ( f K ≈ R{∆ 2 f } ) for 3D implicit surfaces [43,64]. While an inductive bias favoring smoothness is good for reconstructing shapes with dense samples, it is too weak a prior in more challenging cases such as when the input points are very sparse or only cover part of a shape.…”

Section: Inductive Bias Of Neural Splinesmentioning

confidence: 99%